2023-03-20 00:55:20,801 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/39532ada-9017-8fdd-ef0b-a0e859f23e00 2023-03-20 00:55:20,812 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-03-20 00:55:20,824 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-03-20 00:55:20,824 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/39532ada-9017-8fdd-ef0b-a0e859f23e00/cluster_4c1651d3-7e91-3b9e-86c3-2dd8b012a0ed, deleteOnExit=true 2023-03-20 00:55:20,824 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-03-20 00:55:20,825 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/39532ada-9017-8fdd-ef0b-a0e859f23e00/test.cache.data in system properties and HBase conf 2023-03-20 00:55:20,825 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/39532ada-9017-8fdd-ef0b-a0e859f23e00/hadoop.tmp.dir in system properties and HBase conf 2023-03-20 00:55:20,826 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/39532ada-9017-8fdd-ef0b-a0e859f23e00/hadoop.log.dir in system properties and HBase conf 2023-03-20 00:55:20,826 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/39532ada-9017-8fdd-ef0b-a0e859f23e00/mapreduce.cluster.local.dir in system properties and HBase conf 2023-03-20 00:55:20,827 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/39532ada-9017-8fdd-ef0b-a0e859f23e00/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-03-20 00:55:20,827 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-03-20 00:55:20,918 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-03-20 00:55:21,227 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-03-20 00:55:21,232 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/39532ada-9017-8fdd-ef0b-a0e859f23e00/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-03-20 00:55:21,232 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/39532ada-9017-8fdd-ef0b-a0e859f23e00/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-03-20 00:55:21,232 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/39532ada-9017-8fdd-ef0b-a0e859f23e00/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-03-20 00:55:21,233 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/39532ada-9017-8fdd-ef0b-a0e859f23e00/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-20 00:55:21,233 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/39532ada-9017-8fdd-ef0b-a0e859f23e00/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-03-20 00:55:21,233 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/39532ada-9017-8fdd-ef0b-a0e859f23e00/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-03-20 00:55:21,233 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/39532ada-9017-8fdd-ef0b-a0e859f23e00/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-20 00:55:21,234 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/39532ada-9017-8fdd-ef0b-a0e859f23e00/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-20 00:55:21,234 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/39532ada-9017-8fdd-ef0b-a0e859f23e00/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-03-20 00:55:21,234 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/39532ada-9017-8fdd-ef0b-a0e859f23e00/nfs.dump.dir in system properties and HBase conf 2023-03-20 00:55:21,234 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/39532ada-9017-8fdd-ef0b-a0e859f23e00/java.io.tmpdir in system properties and HBase conf 2023-03-20 00:55:21,235 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/39532ada-9017-8fdd-ef0b-a0e859f23e00/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-20 00:55:21,235 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/39532ada-9017-8fdd-ef0b-a0e859f23e00/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-03-20 00:55:21,235 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/39532ada-9017-8fdd-ef0b-a0e859f23e00/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-03-20 00:55:21,637 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-20 00:55:21,641 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-20 00:55:21,861 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-03-20 00:55:22,002 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-03-20 00:55:22,014 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-20 00:55:22,046 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-03-20 00:55:22,082 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/39532ada-9017-8fdd-ef0b-a0e859f23e00/java.io.tmpdir/Jetty_localhost_localdomain_34743_hdfs____wytoen/webapp 2023-03-20 00:55:22,185 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:34743 2023-03-20 00:55:22,217 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-20 00:55:22,218 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-20 00:55:22,539 WARN [Listener at localhost.localdomain/35057] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-20 00:55:22,625 WARN [Listener at localhost.localdomain/35057] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-03-20 00:55:22,639 WARN [Listener at localhost.localdomain/35057] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-20 00:55:22,644 INFO [Listener at localhost.localdomain/35057] log.Slf4jLog(67): jetty-6.1.26 2023-03-20 00:55:22,649 INFO [Listener at localhost.localdomain/35057] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/39532ada-9017-8fdd-ef0b-a0e859f23e00/java.io.tmpdir/Jetty_localhost_44229_datanode____.v5ujhc/webapp 2023-03-20 00:55:22,730 INFO [Listener at localhost.localdomain/35057] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44229 2023-03-20 00:55:23,024 WARN [Listener at localhost.localdomain/43361] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-20 00:55:23,361 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd4b9937206e459c2: Processing first storage report for DS-a91e9420-d499-4be8-aecb-966a66b7b679 from datanode 14d682db-b1e5-40dd-85c3-3fd110b80fa0 2023-03-20 00:55:23,362 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd4b9937206e459c2: from storage DS-a91e9420-d499-4be8-aecb-966a66b7b679 node DatanodeRegistration(127.0.0.1:42335, datanodeUuid=14d682db-b1e5-40dd-85c3-3fd110b80fa0, infoPort=33807, infoSecurePort=0, ipcPort=43361, storageInfo=lv=-57;cid=testClusterID;nsid=761387553;c=1679273721698), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-03-20 00:55:23,363 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd4b9937206e459c2: Processing first storage report for DS-b3d1919b-95c1-4b7b-ad12-67af636428f0 from datanode 14d682db-b1e5-40dd-85c3-3fd110b80fa0 2023-03-20 00:55:23,363 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd4b9937206e459c2: from storage DS-b3d1919b-95c1-4b7b-ad12-67af636428f0 node DatanodeRegistration(127.0.0.1:42335, datanodeUuid=14d682db-b1e5-40dd-85c3-3fd110b80fa0, infoPort=33807, infoSecurePort=0, ipcPort=43361, storageInfo=lv=-57;cid=testClusterID;nsid=761387553;c=1679273721698), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-03-20 00:55:23,458 DEBUG [Listener at localhost.localdomain/43361] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/39532ada-9017-8fdd-ef0b-a0e859f23e00 2023-03-20 00:55:23,515 INFO [Listener at localhost.localdomain/43361] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/39532ada-9017-8fdd-ef0b-a0e859f23e00/cluster_4c1651d3-7e91-3b9e-86c3-2dd8b012a0ed/zookeeper_0, clientPort=60547, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/39532ada-9017-8fdd-ef0b-a0e859f23e00/cluster_4c1651d3-7e91-3b9e-86c3-2dd8b012a0ed/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/39532ada-9017-8fdd-ef0b-a0e859f23e00/cluster_4c1651d3-7e91-3b9e-86c3-2dd8b012a0ed/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-03-20 00:55:23,526 INFO [Listener at localhost.localdomain/43361] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=60547 2023-03-20 00:55:23,532 INFO [Listener at localhost.localdomain/43361] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-20 00:55:23,534 INFO [Listener at localhost.localdomain/43361] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-20 00:55:24,149 INFO [Listener at localhost.localdomain/43361] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6 with version=8 2023-03-20 00:55:24,149 INFO [Listener at localhost.localdomain/43361] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/hbase-staging 2023-03-20 00:55:24,410 INFO [Listener at localhost.localdomain/43361] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-20 00:55:24,766 INFO [Listener at localhost.localdomain/43361] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-03-20 00:55:24,791 INFO [Listener at localhost.localdomain/43361] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-20 00:55:24,791 INFO [Listener at localhost.localdomain/43361] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-20 00:55:24,791 INFO [Listener at localhost.localdomain/43361] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-20 00:55:24,791 INFO [Listener at localhost.localdomain/43361] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-20 00:55:24,792 INFO [Listener at localhost.localdomain/43361] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-20 00:55:24,905 INFO [Listener at localhost.localdomain/43361] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-03-20 00:55:24,963 DEBUG [Listener at localhost.localdomain/43361] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-03-20 00:55:25,041 INFO [Listener at localhost.localdomain/43361] ipc.NettyRpcServer(120): Bind to /148.251.75.209:44395 2023-03-20 00:55:25,051 INFO [Listener at localhost.localdomain/43361] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-20 00:55:25,053 INFO [Listener at localhost.localdomain/43361] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-20 00:55:25,069 INFO [Listener at localhost.localdomain/43361] zookeeper.RecoverableZooKeeper(93): Process identifier=master:44395 connecting to ZooKeeper ensemble=127.0.0.1:60547 2023-03-20 00:55:25,098 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:443950x0, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-20 00:55:25,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:44395-0x1002c399e9c0000 connected 2023-03-20 00:55:25,127 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-20 00:55:25,128 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-20 00:55:25,131 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-20 00:55:25,139 DEBUG [Listener at localhost.localdomain/43361] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=44395 2023-03-20 00:55:25,139 DEBUG [Listener at localhost.localdomain/43361] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=44395 2023-03-20 00:55:25,140 DEBUG [Listener at localhost.localdomain/43361] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=44395 2023-03-20 00:55:25,140 DEBUG [Listener at localhost.localdomain/43361] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=44395 2023-03-20 00:55:25,140 DEBUG [Listener at localhost.localdomain/43361] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=44395 2023-03-20 00:55:25,145 INFO [Listener at localhost.localdomain/43361] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6, hbase.cluster.distributed=false 2023-03-20 00:55:25,200 INFO [Listener at localhost.localdomain/43361] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-03-20 00:55:25,200 INFO [Listener at localhost.localdomain/43361] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-20 00:55:25,200 INFO [Listener at localhost.localdomain/43361] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-20 00:55:25,200 INFO [Listener at localhost.localdomain/43361] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-20 00:55:25,200 INFO [Listener at localhost.localdomain/43361] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-20 00:55:25,200 INFO [Listener at localhost.localdomain/43361] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-20 00:55:25,204 INFO [Listener at localhost.localdomain/43361] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-03-20 00:55:25,207 INFO [Listener at localhost.localdomain/43361] ipc.NettyRpcServer(120): Bind to /148.251.75.209:34553 2023-03-20 00:55:25,209 INFO [Listener at localhost.localdomain/43361] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-03-20 00:55:25,213 DEBUG [Listener at localhost.localdomain/43361] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-03-20 00:55:25,214 INFO [Listener at localhost.localdomain/43361] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-20 00:55:25,217 INFO [Listener at localhost.localdomain/43361] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-20 00:55:25,218 INFO [Listener at localhost.localdomain/43361] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34553 connecting to ZooKeeper ensemble=127.0.0.1:60547 2023-03-20 00:55:25,221 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:345530x0, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-20 00:55:25,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:34553-0x1002c399e9c0001 connected 2023-03-20 00:55:25,222 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-20 00:55:25,223 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-20 00:55:25,224 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-20 00:55:25,224 DEBUG [Listener at localhost.localdomain/43361] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34553 2023-03-20 00:55:25,224 DEBUG [Listener at localhost.localdomain/43361] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34553 2023-03-20 00:55:25,225 DEBUG [Listener at localhost.localdomain/43361] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34553 2023-03-20 00:55:25,225 DEBUG [Listener at localhost.localdomain/43361] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34553 2023-03-20 00:55:25,225 DEBUG [Listener at localhost.localdomain/43361] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34553 2023-03-20 00:55:25,227 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,44395,1679273724282 2023-03-20 00:55:25,237 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-20 00:55:25,239 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,44395,1679273724282 2023-03-20 00:55:25,255 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-20 00:55:25,255 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-20 00:55:25,255 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-20 00:55:25,256 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-20 00:55:25,257 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,44395,1679273724282 from backup master directory 2023-03-20 00:55:25,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-20 00:55:25,260 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,44395,1679273724282 2023-03-20 00:55:25,260 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-20 00:55:25,260 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-20 00:55:25,260 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,44395,1679273724282 2023-03-20 00:55:25,263 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-03-20 00:55:25,264 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-03-20 00:55:25,338 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/hbase.id with ID: 8444fb19-73eb-4f30-b8ff-ab35840d2eff 2023-03-20 00:55:25,377 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-20 00:55:25,391 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-20 00:55:25,433 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x09c7f582 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:55:25,462 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1859153d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:55:25,481 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-20 00:55:25,482 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-03-20 00:55:25,496 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-03-20 00:55:25,496 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-03-20 00:55:25,498 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-20 00:55:25,501 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-20 00:55:25,502 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-20 00:55:25,527 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/MasterData/data/master/store-tmp 2023-03-20 00:55:25,550 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 00:55:25,550 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-20 00:55:25,550 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-20 00:55:25,550 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-20 00:55:25,550 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-20 00:55:25,550 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-20 00:55:25,550 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-20 00:55:25,550 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-20 00:55:25,552 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/MasterData/WALs/jenkins-hbase20.apache.org,44395,1679273724282 2023-03-20 00:55:25,575 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C44395%2C1679273724282, suffix=, logDir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/MasterData/WALs/jenkins-hbase20.apache.org,44395,1679273724282, archiveDir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/MasterData/oldWALs, maxLogs=10 2023-03-20 00:55:25,620 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42335,DS-a91e9420-d499-4be8-aecb-966a66b7b679,DISK] 2023-03-20 00:55:25,626 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 00:55:25,676 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/MasterData/WALs/jenkins-hbase20.apache.org,44395,1679273724282/jenkins-hbase20.apache.org%2C44395%2C1679273724282.1679273725583 2023-03-20 00:55:25,677 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42335,DS-a91e9420-d499-4be8-aecb-966a66b7b679,DISK]] 2023-03-20 00:55:25,678 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-03-20 00:55:25,678 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 00:55:25,680 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-03-20 00:55:25,681 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-03-20 00:55:25,728 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-03-20 00:55:25,734 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-03-20 00:55:25,753 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-03-20 00:55:25,767 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:55:25,773 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-20 00:55:25,775 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-20 00:55:25,790 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-03-20 00:55:25,795 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-20 00:55:25,796 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74695473, jitterRate=0.11304928362369537}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-20 00:55:25,796 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-20 00:55:25,797 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-03-20 00:55:25,813 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-03-20 00:55:25,813 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-03-20 00:55:25,815 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-03-20 00:55:25,817 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-03-20 00:55:25,844 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 27 msec 2023-03-20 00:55:25,845 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-03-20 00:55:25,865 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-03-20 00:55:25,870 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-03-20 00:55:25,890 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-03-20 00:55:25,893 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-20 00:55:25,894 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-03-20 00:55:25,899 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-03-20 00:55:25,902 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-03-20 00:55:25,905 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-20 00:55:25,906 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-03-20 00:55:25,907 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-03-20 00:55:25,917 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-03-20 00:55:25,921 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-20 00:55:25,921 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-20 00:55:25,921 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-20 00:55:25,921 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,44395,1679273724282, sessionid=0x1002c399e9c0000, setting cluster-up flag (Was=false) 2023-03-20 00:55:25,938 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-20 00:55:25,941 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-03-20 00:55:25,943 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,44395,1679273724282 2023-03-20 00:55:25,959 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-20 00:55:25,963 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-03-20 00:55:25,965 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,44395,1679273724282 2023-03-20 00:55:25,967 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.hbase-snapshot/.tmp 2023-03-20 00:55:26,029 INFO [RS:0;jenkins-hbase20:34553] regionserver.HRegionServer(952): ClusterId : 8444fb19-73eb-4f30-b8ff-ab35840d2eff 2023-03-20 00:55:26,033 DEBUG [RS:0;jenkins-hbase20:34553] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-03-20 00:55:26,038 DEBUG [RS:0;jenkins-hbase20:34553] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-03-20 00:55:26,039 DEBUG [RS:0;jenkins-hbase20:34553] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-03-20 00:55:26,044 DEBUG [RS:0;jenkins-hbase20:34553] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-03-20 00:55:26,046 DEBUG [RS:0;jenkins-hbase20:34553] zookeeper.ReadOnlyZKClient(139): Connect 0x01e536ae to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:55:26,054 DEBUG [RS:0;jenkins-hbase20:34553] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@f22c8e2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:55:26,055 DEBUG [RS:0;jenkins-hbase20:34553] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b1d5dde, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-03-20 00:55:26,069 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-03-20 00:55:26,082 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-03-20 00:55:26,082 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-03-20 00:55:26,082 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-03-20 00:55:26,083 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-03-20 00:55:26,083 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-03-20 00:55:26,083 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-20 00:55:26,083 DEBUG [RS:0;jenkins-hbase20:34553] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase20:34553 2023-03-20 00:55:26,083 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-03-20 00:55:26,083 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-20 00:55:26,086 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1679273756086 2023-03-20 00:55:26,087 INFO [RS:0;jenkins-hbase20:34553] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-03-20 00:55:26,088 INFO [RS:0;jenkins-hbase20:34553] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-03-20 00:55:26,088 DEBUG [RS:0;jenkins-hbase20:34553] regionserver.HRegionServer(1023): About to register with Master. 2023-03-20 00:55:26,088 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-03-20 00:55:26,090 INFO [RS:0;jenkins-hbase20:34553] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase20.apache.org,44395,1679273724282 with isa=jenkins-hbase20.apache.org/148.251.75.209:34553, startcode=1679273725199 2023-03-20 00:55:26,091 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-03-20 00:55:26,091 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-03-20 00:55:26,096 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-20 00:55:26,099 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-03-20 00:55:26,104 DEBUG [RS:0;jenkins-hbase20:34553] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-03-20 00:55:26,105 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-03-20 00:55:26,105 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-03-20 00:55:26,106 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-03-20 00:55:26,106 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-03-20 00:55:26,111 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-20 00:55:26,112 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-03-20 00:55:26,114 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-03-20 00:55:26,115 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-03-20 00:55:26,118 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-03-20 00:55:26,120 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-03-20 00:55:26,124 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1679273726122,5,FailOnTimeoutGroup] 2023-03-20 00:55:26,124 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1679273726124,5,FailOnTimeoutGroup] 2023-03-20 00:55:26,124 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-20 00:55:26,125 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-03-20 00:55:26,126 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-03-20 00:55:26,126 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-03-20 00:55:26,152 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:37359, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-03-20 00:55:26,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:26,177 DEBUG [RS:0;jenkins-hbase20:34553] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6 2023-03-20 00:55:26,177 DEBUG [RS:0;jenkins-hbase20:34553] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:35057 2023-03-20 00:55:26,177 DEBUG [RS:0;jenkins-hbase20:34553] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-03-20 00:55:26,181 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-20 00:55:26,182 DEBUG [RS:0;jenkins-hbase20:34553] zookeeper.ZKUtil(162): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:26,182 WARN [RS:0;jenkins-hbase20:34553] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-20 00:55:26,183 INFO [RS:0;jenkins-hbase20:34553] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-20 00:55:26,183 DEBUG [RS:0;jenkins-hbase20:34553] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/WALs/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:26,184 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,34553,1679273725199] 2023-03-20 00:55:26,194 DEBUG [RS:0;jenkins-hbase20:34553] zookeeper.ZKUtil(162): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:26,204 DEBUG [RS:0;jenkins-hbase20:34553] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-03-20 00:55:26,211 INFO [RS:0;jenkins-hbase20:34553] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-03-20 00:55:26,227 INFO [RS:0;jenkins-hbase20:34553] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-03-20 00:55:26,232 INFO [RS:0;jenkins-hbase20:34553] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-03-20 00:55:26,232 INFO [RS:0;jenkins-hbase20:34553] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-03-20 00:55:26,233 INFO [RS:0;jenkins-hbase20:34553] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-03-20 00:55:26,239 INFO [RS:0;jenkins-hbase20:34553] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-03-20 00:55:26,239 DEBUG [RS:0;jenkins-hbase20:34553] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-20 00:55:26,239 DEBUG [RS:0;jenkins-hbase20:34553] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-20 00:55:26,239 DEBUG [RS:0;jenkins-hbase20:34553] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-20 00:55:26,240 DEBUG [RS:0;jenkins-hbase20:34553] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-20 00:55:26,240 DEBUG [RS:0;jenkins-hbase20:34553] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-20 00:55:26,240 DEBUG [RS:0;jenkins-hbase20:34553] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-03-20 00:55:26,240 DEBUG [RS:0;jenkins-hbase20:34553] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-20 00:55:26,240 DEBUG [RS:0;jenkins-hbase20:34553] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-20 00:55:26,240 DEBUG [RS:0;jenkins-hbase20:34553] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-20 00:55:26,240 DEBUG [RS:0;jenkins-hbase20:34553] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-20 00:55:26,241 INFO [RS:0;jenkins-hbase20:34553] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-03-20 00:55:26,241 INFO [RS:0;jenkins-hbase20:34553] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-03-20 00:55:26,241 INFO [RS:0;jenkins-hbase20:34553] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-03-20 00:55:26,254 INFO [RS:0;jenkins-hbase20:34553] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-03-20 00:55:26,256 INFO [RS:0;jenkins-hbase20:34553] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,34553,1679273725199-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-20 00:55:26,269 INFO [RS:0;jenkins-hbase20:34553] regionserver.Replication(203): jenkins-hbase20.apache.org,34553,1679273725199 started 2023-03-20 00:55:26,269 INFO [RS:0;jenkins-hbase20:34553] regionserver.HRegionServer(1638): Serving as jenkins-hbase20.apache.org,34553,1679273725199, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:34553, sessionid=0x1002c399e9c0001 2023-03-20 00:55:26,269 DEBUG [RS:0;jenkins-hbase20:34553] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-03-20 00:55:26,269 DEBUG [RS:0;jenkins-hbase20:34553] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:26,269 DEBUG [RS:0;jenkins-hbase20:34553] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,34553,1679273725199' 2023-03-20 00:55:26,269 DEBUG [RS:0;jenkins-hbase20:34553] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:26,270 DEBUG [RS:0;jenkins-hbase20:34553] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:26,271 DEBUG [RS:0;jenkins-hbase20:34553] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-03-20 00:55:26,271 DEBUG [RS:0;jenkins-hbase20:34553] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-03-20 00:55:26,271 DEBUG [RS:0;jenkins-hbase20:34553] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:26,271 DEBUG [RS:0;jenkins-hbase20:34553] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,34553,1679273725199' 2023-03-20 00:55:26,271 DEBUG [RS:0;jenkins-hbase20:34553] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-03-20 00:55:26,271 DEBUG [RS:0;jenkins-hbase20:34553] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-03-20 00:55:26,272 DEBUG [RS:0;jenkins-hbase20:34553] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-03-20 00:55:26,272 INFO [RS:0;jenkins-hbase20:34553] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-03-20 00:55:26,272 INFO [RS:0;jenkins-hbase20:34553] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-03-20 00:55:26,387 INFO [RS:0;jenkins-hbase20:34553] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C34553%2C1679273725199, suffix=, logDir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/WALs/jenkins-hbase20.apache.org,34553,1679273725199, archiveDir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/oldWALs, maxLogs=32 2023-03-20 00:55:26,403 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42335,DS-a91e9420-d499-4be8-aecb-966a66b7b679,DISK] 2023-03-20 00:55:26,406 INFO [RS:0;jenkins-hbase20:34553] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/WALs/jenkins-hbase20.apache.org,34553,1679273725199/jenkins-hbase20.apache.org%2C34553%2C1679273725199.1679273726389 2023-03-20 00:55:26,406 DEBUG [RS:0;jenkins-hbase20:34553] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42335,DS-a91e9420-d499-4be8-aecb-966a66b7b679,DISK]] 2023-03-20 00:55:26,530 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-20 00:55:26,531 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-20 00:55:26,532 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6 2023-03-20 00:55:26,953 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 00:55:26,955 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-20 00:55:26,958 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/meta/1588230740/info 2023-03-20 00:55:26,958 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-20 00:55:26,960 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:55:26,960 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-20 00:55:26,962 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/meta/1588230740/rep_barrier 2023-03-20 00:55:26,963 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-20 00:55:26,963 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:55:26,964 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-20 00:55:26,965 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/meta/1588230740/table 2023-03-20 00:55:26,966 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-20 00:55:26,967 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:55:26,968 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/meta/1588230740 2023-03-20 00:55:26,969 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/meta/1588230740 2023-03-20 00:55:26,972 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-20 00:55:26,974 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-20 00:55:26,977 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-20 00:55:26,978 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69723221, jitterRate=0.03895695507526398}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-20 00:55:26,978 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-20 00:55:26,978 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-20 00:55:26,978 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-20 00:55:26,979 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-20 00:55:26,979 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-20 00:55:26,979 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-20 00:55:26,979 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-20 00:55:26,980 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-20 00:55:26,985 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-03-20 00:55:26,985 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-03-20 00:55:26,992 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-03-20 00:55:27,002 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-03-20 00:55:27,004 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-03-20 00:55:27,157 DEBUG [jenkins-hbase20:44395] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-03-20 00:55:27,160 INFO [PEWorker-2] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,34553,1679273725199, state=OPENING 2023-03-20 00:55:27,168 DEBUG [PEWorker-2] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-03-20 00:55:27,170 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-20 00:55:27,170 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-20 00:55:27,173 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,34553,1679273725199}] 2023-03-20 00:55:27,357 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:27,359 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-03-20 00:55:27,364 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:37864, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-03-20 00:55:27,379 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-03-20 00:55:27,379 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-20 00:55:27,384 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C34553%2C1679273725199.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/WALs/jenkins-hbase20.apache.org,34553,1679273725199, archiveDir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/oldWALs, maxLogs=32 2023-03-20 00:55:27,399 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42335,DS-a91e9420-d499-4be8-aecb-966a66b7b679,DISK] 2023-03-20 00:55:27,403 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/WALs/jenkins-hbase20.apache.org,34553,1679273725199/jenkins-hbase20.apache.org%2C34553%2C1679273725199.meta.1679273727385.meta 2023-03-20 00:55:27,403 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42335,DS-a91e9420-d499-4be8-aecb-966a66b7b679,DISK]] 2023-03-20 00:55:27,404 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-03-20 00:55:27,405 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-03-20 00:55:27,419 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-03-20 00:55:27,423 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-03-20 00:55:27,427 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-03-20 00:55:27,427 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 00:55:27,428 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-03-20 00:55:27,428 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-03-20 00:55:27,430 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-20 00:55:27,432 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/meta/1588230740/info 2023-03-20 00:55:27,432 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/meta/1588230740/info 2023-03-20 00:55:27,433 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-20 00:55:27,434 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:55:27,434 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-20 00:55:27,436 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/meta/1588230740/rep_barrier 2023-03-20 00:55:27,436 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/meta/1588230740/rep_barrier 2023-03-20 00:55:27,436 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-20 00:55:27,437 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:55:27,437 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-20 00:55:27,438 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/meta/1588230740/table 2023-03-20 00:55:27,439 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/meta/1588230740/table 2023-03-20 00:55:27,439 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-20 00:55:27,440 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:55:27,441 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/meta/1588230740 2023-03-20 00:55:27,444 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/meta/1588230740 2023-03-20 00:55:27,447 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-20 00:55:27,449 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-20 00:55:27,451 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67917718, jitterRate=0.012052863836288452}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-20 00:55:27,451 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-20 00:55:27,461 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1679273727351 2023-03-20 00:55:27,476 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-03-20 00:55:27,477 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-03-20 00:55:27,477 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,34553,1679273725199, state=OPEN 2023-03-20 00:55:27,479 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-03-20 00:55:27,479 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-20 00:55:27,483 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-03-20 00:55:27,483 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,34553,1679273725199 in 306 msec 2023-03-20 00:55:27,488 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-03-20 00:55:27,488 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 492 msec 2023-03-20 00:55:27,493 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.4870 sec 2023-03-20 00:55:27,494 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1679273727493, completionTime=-1 2023-03-20 00:55:27,494 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-03-20 00:55:27,494 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-03-20 00:55:27,559 DEBUG [hconnection-0x5f8fccde-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:55:27,563 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:37870, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:55:27,584 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-03-20 00:55:27,584 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1679273787584 2023-03-20 00:55:27,584 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1679273847584 2023-03-20 00:55:27,584 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 90 msec 2023-03-20 00:55:27,618 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,44395,1679273724282-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-20 00:55:27,619 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,44395,1679273724282-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-20 00:55:27,619 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,44395,1679273724282-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-20 00:55:27,621 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:44395, period=300000, unit=MILLISECONDS is enabled. 2023-03-20 00:55:27,621 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-03-20 00:55:27,627 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-03-20 00:55:27,639 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-03-20 00:55:27,640 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-20 00:55:27,652 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-03-20 00:55:27,654 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-03-20 00:55:27,657 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-20 00:55:27,676 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/hbase/namespace/eae49a7cbb97a0b15515bb68f94a4bb9 2023-03-20 00:55:27,678 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/hbase/namespace/eae49a7cbb97a0b15515bb68f94a4bb9 empty. 2023-03-20 00:55:27,679 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/hbase/namespace/eae49a7cbb97a0b15515bb68f94a4bb9 2023-03-20 00:55:27,679 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-03-20 00:55:27,704 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-03-20 00:55:27,706 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => eae49a7cbb97a0b15515bb68f94a4bb9, NAME => 'hbase:namespace,,1679273727640.eae49a7cbb97a0b15515bb68f94a4bb9.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp 2023-03-20 00:55:28,120 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1679273727640.eae49a7cbb97a0b15515bb68f94a4bb9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 00:55:28,121 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing eae49a7cbb97a0b15515bb68f94a4bb9, disabling compactions & flushes 2023-03-20 00:55:28,121 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1679273727640.eae49a7cbb97a0b15515bb68f94a4bb9. 2023-03-20 00:55:28,121 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1679273727640.eae49a7cbb97a0b15515bb68f94a4bb9. 2023-03-20 00:55:28,122 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1679273727640.eae49a7cbb97a0b15515bb68f94a4bb9. after waiting 0 ms 2023-03-20 00:55:28,122 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1679273727640.eae49a7cbb97a0b15515bb68f94a4bb9. 2023-03-20 00:55:28,122 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1679273727640.eae49a7cbb97a0b15515bb68f94a4bb9. 2023-03-20 00:55:28,122 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for eae49a7cbb97a0b15515bb68f94a4bb9: 2023-03-20 00:55:28,130 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-03-20 00:55:28,143 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1679273727640.eae49a7cbb97a0b15515bb68f94a4bb9.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679273728133"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679273728133"}]},"ts":"1679273728133"} 2023-03-20 00:55:28,166 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-20 00:55:28,168 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-20 00:55:28,172 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679273728168"}]},"ts":"1679273728168"} 2023-03-20 00:55:28,177 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-03-20 00:55:28,185 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=eae49a7cbb97a0b15515bb68f94a4bb9, ASSIGN}] 2023-03-20 00:55:28,188 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=eae49a7cbb97a0b15515bb68f94a4bb9, ASSIGN 2023-03-20 00:55:28,190 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=eae49a7cbb97a0b15515bb68f94a4bb9, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34553,1679273725199; forceNewPlan=false, retain=false 2023-03-20 00:55:28,342 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=eae49a7cbb97a0b15515bb68f94a4bb9, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:28,342 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1679273727640.eae49a7cbb97a0b15515bb68f94a4bb9.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679273728341"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679273728341"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679273728341"}]},"ts":"1679273728341"} 2023-03-20 00:55:28,347 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure eae49a7cbb97a0b15515bb68f94a4bb9, server=jenkins-hbase20.apache.org,34553,1679273725199}] 2023-03-20 00:55:28,511 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1679273727640.eae49a7cbb97a0b15515bb68f94a4bb9. 2023-03-20 00:55:28,512 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => eae49a7cbb97a0b15515bb68f94a4bb9, NAME => 'hbase:namespace,,1679273727640.eae49a7cbb97a0b15515bb68f94a4bb9.', STARTKEY => '', ENDKEY => ''} 2023-03-20 00:55:28,513 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace eae49a7cbb97a0b15515bb68f94a4bb9 2023-03-20 00:55:28,514 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1679273727640.eae49a7cbb97a0b15515bb68f94a4bb9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 00:55:28,514 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for eae49a7cbb97a0b15515bb68f94a4bb9 2023-03-20 00:55:28,514 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for eae49a7cbb97a0b15515bb68f94a4bb9 2023-03-20 00:55:28,517 INFO [StoreOpener-eae49a7cbb97a0b15515bb68f94a4bb9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region eae49a7cbb97a0b15515bb68f94a4bb9 2023-03-20 00:55:28,520 DEBUG [StoreOpener-eae49a7cbb97a0b15515bb68f94a4bb9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/namespace/eae49a7cbb97a0b15515bb68f94a4bb9/info 2023-03-20 00:55:28,520 DEBUG [StoreOpener-eae49a7cbb97a0b15515bb68f94a4bb9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/namespace/eae49a7cbb97a0b15515bb68f94a4bb9/info 2023-03-20 00:55:28,521 INFO [StoreOpener-eae49a7cbb97a0b15515bb68f94a4bb9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region eae49a7cbb97a0b15515bb68f94a4bb9 columnFamilyName info 2023-03-20 00:55:28,522 INFO [StoreOpener-eae49a7cbb97a0b15515bb68f94a4bb9-1] regionserver.HStore(310): Store=eae49a7cbb97a0b15515bb68f94a4bb9/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:55:28,524 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/namespace/eae49a7cbb97a0b15515bb68f94a4bb9 2023-03-20 00:55:28,525 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/namespace/eae49a7cbb97a0b15515bb68f94a4bb9 2023-03-20 00:55:28,530 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for eae49a7cbb97a0b15515bb68f94a4bb9 2023-03-20 00:55:28,534 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/namespace/eae49a7cbb97a0b15515bb68f94a4bb9/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-20 00:55:28,534 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened eae49a7cbb97a0b15515bb68f94a4bb9; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69619885, jitterRate=0.037417128682136536}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-20 00:55:28,535 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for eae49a7cbb97a0b15515bb68f94a4bb9: 2023-03-20 00:55:28,536 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1679273727640.eae49a7cbb97a0b15515bb68f94a4bb9., pid=6, masterSystemTime=1679273728500 2023-03-20 00:55:28,540 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1679273727640.eae49a7cbb97a0b15515bb68f94a4bb9. 2023-03-20 00:55:28,540 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1679273727640.eae49a7cbb97a0b15515bb68f94a4bb9. 2023-03-20 00:55:28,542 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=eae49a7cbb97a0b15515bb68f94a4bb9, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:28,542 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1679273727640.eae49a7cbb97a0b15515bb68f94a4bb9.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679273728541"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679273728541"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679273728541"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679273728541"}]},"ts":"1679273728541"} 2023-03-20 00:55:28,551 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-03-20 00:55:28,551 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure eae49a7cbb97a0b15515bb68f94a4bb9, server=jenkins-hbase20.apache.org,34553,1679273725199 in 200 msec 2023-03-20 00:55:28,554 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-03-20 00:55:28,555 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=eae49a7cbb97a0b15515bb68f94a4bb9, ASSIGN in 366 msec 2023-03-20 00:55:28,556 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-20 00:55:28,556 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679273728556"}]},"ts":"1679273728556"} 2023-03-20 00:55:28,559 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-03-20 00:55:28,562 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-03-20 00:55:28,564 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 920 msec 2023-03-20 00:55:28,656 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-03-20 00:55:28,658 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-03-20 00:55:28,658 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-20 00:55:28,700 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-03-20 00:55:28,719 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-20 00:55:28,725 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 34 msec 2023-03-20 00:55:28,735 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-03-20 00:55:28,748 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-20 00:55:28,753 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 18 msec 2023-03-20 00:55:28,762 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-03-20 00:55:28,764 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-03-20 00:55:28,764 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.504sec 2023-03-20 00:55:28,766 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-03-20 00:55:28,768 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-03-20 00:55:28,768 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-03-20 00:55:28,769 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,44395,1679273724282-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-03-20 00:55:28,771 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,44395,1679273724282-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-03-20 00:55:28,781 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-03-20 00:55:28,839 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x6a972b34 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:55:28,844 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6dc6e9d6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:55:28,856 DEBUG [hconnection-0x3fa88829-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:55:28,866 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:37878, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:55:28,875 INFO [Listener at localhost.localdomain/43361] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase20.apache.org,44395,1679273724282 2023-03-20 00:55:28,892 INFO [Listener at localhost.localdomain/43361] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=207, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=267, ProcessCount=170, AvailableMemoryMB=5126 2023-03-20 00:55:28,906 DEBUG [Listener at localhost.localdomain/43361] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-20 00:55:28,909 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:51732, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-20 00:55:28,917 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-20 00:55:28,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-20 00:55:28,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-20 00:55:28,926 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-20 00:55:28,928 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-20 00:55:28,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-03-20 00:55:28,931 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:28,933 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98 empty. 2023-03-20 00:55:28,934 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:28,934 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-20 00:55:28,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-20 00:55:28,953 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-20 00:55:28,955 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 1bfc994bd080bb1b8cbc4aef89c8cc98, NAME => 'TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp 2023-03-20 00:55:29,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-20 00:55:29,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-20 00:55:29,370 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 00:55:29,371 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 1bfc994bd080bb1b8cbc4aef89c8cc98, disabling compactions & flushes 2023-03-20 00:55:29,372 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:29,372 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:29,372 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. after waiting 0 ms 2023-03-20 00:55:29,372 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:29,372 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:29,373 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:29,381 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-20 00:55:29,383 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273729382"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679273729382"}]},"ts":"1679273729382"} 2023-03-20 00:55:29,386 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-20 00:55:29,387 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-20 00:55:29,388 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679273729387"}]},"ts":"1679273729387"} 2023-03-20 00:55:29,390 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-20 00:55:29,394 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1bfc994bd080bb1b8cbc4aef89c8cc98, ASSIGN}] 2023-03-20 00:55:29,396 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1bfc994bd080bb1b8cbc4aef89c8cc98, ASSIGN 2023-03-20 00:55:29,397 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=1bfc994bd080bb1b8cbc4aef89c8cc98, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34553,1679273725199; forceNewPlan=false, retain=false 2023-03-20 00:55:29,549 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=1bfc994bd080bb1b8cbc4aef89c8cc98, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:29,550 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273729549"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679273729549"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679273729549"}]},"ts":"1679273729549"} 2023-03-20 00:55:29,556 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199}] 2023-03-20 00:55:29,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-20 00:55:29,724 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:29,725 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1bfc994bd080bb1b8cbc4aef89c8cc98, NAME => 'TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.', STARTKEY => '', ENDKEY => ''} 2023-03-20 00:55:29,725 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:29,725 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 00:55:29,726 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:29,726 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:29,729 INFO [StoreOpener-1bfc994bd080bb1b8cbc4aef89c8cc98-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:29,732 DEBUG [StoreOpener-1bfc994bd080bb1b8cbc4aef89c8cc98-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A 2023-03-20 00:55:29,732 DEBUG [StoreOpener-1bfc994bd080bb1b8cbc4aef89c8cc98-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A 2023-03-20 00:55:29,735 INFO [StoreOpener-1bfc994bd080bb1b8cbc4aef89c8cc98-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 00:55:29,735 INFO [StoreOpener-1bfc994bd080bb1b8cbc4aef89c8cc98-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1bfc994bd080bb1b8cbc4aef89c8cc98 columnFamilyName A 2023-03-20 00:55:29,737 INFO [StoreOpener-1bfc994bd080bb1b8cbc4aef89c8cc98-1] regionserver.HStore(310): Store=1bfc994bd080bb1b8cbc4aef89c8cc98/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:55:29,737 INFO [StoreOpener-1bfc994bd080bb1b8cbc4aef89c8cc98-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:29,739 DEBUG [StoreOpener-1bfc994bd080bb1b8cbc4aef89c8cc98-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B 2023-03-20 00:55:29,740 DEBUG [StoreOpener-1bfc994bd080bb1b8cbc4aef89c8cc98-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B 2023-03-20 00:55:29,740 INFO [StoreOpener-1bfc994bd080bb1b8cbc4aef89c8cc98-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 00:55:29,741 INFO [StoreOpener-1bfc994bd080bb1b8cbc4aef89c8cc98-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1bfc994bd080bb1b8cbc4aef89c8cc98 columnFamilyName B 2023-03-20 00:55:29,742 INFO [StoreOpener-1bfc994bd080bb1b8cbc4aef89c8cc98-1] regionserver.HStore(310): Store=1bfc994bd080bb1b8cbc4aef89c8cc98/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:55:29,742 INFO [StoreOpener-1bfc994bd080bb1b8cbc4aef89c8cc98-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:29,744 DEBUG [StoreOpener-1bfc994bd080bb1b8cbc4aef89c8cc98-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C 2023-03-20 00:55:29,745 DEBUG [StoreOpener-1bfc994bd080bb1b8cbc4aef89c8cc98-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C 2023-03-20 00:55:29,745 INFO [StoreOpener-1bfc994bd080bb1b8cbc4aef89c8cc98-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 00:55:29,745 INFO [StoreOpener-1bfc994bd080bb1b8cbc4aef89c8cc98-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1bfc994bd080bb1b8cbc4aef89c8cc98 columnFamilyName C 2023-03-20 00:55:29,747 INFO [StoreOpener-1bfc994bd080bb1b8cbc4aef89c8cc98-1] regionserver.HStore(310): Store=1bfc994bd080bb1b8cbc4aef89c8cc98/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:55:29,748 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:29,751 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:29,752 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:29,763 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-20 00:55:29,766 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:29,771 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-20 00:55:29,772 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1bfc994bd080bb1b8cbc4aef89c8cc98; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72185643, jitterRate=0.07564990222454071}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-20 00:55:29,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:29,773 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98., pid=11, masterSystemTime=1679273729713 2023-03-20 00:55:29,776 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:29,776 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:29,777 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=1bfc994bd080bb1b8cbc4aef89c8cc98, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:29,777 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273729777"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679273729777"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679273729777"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679273729777"}]},"ts":"1679273729777"} 2023-03-20 00:55:29,785 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-03-20 00:55:29,785 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 in 225 msec 2023-03-20 00:55:29,790 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-03-20 00:55:29,792 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1bfc994bd080bb1b8cbc4aef89c8cc98, ASSIGN in 391 msec 2023-03-20 00:55:29,793 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-20 00:55:29,793 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679273729793"}]},"ts":"1679273729793"} 2023-03-20 00:55:29,796 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-20 00:55:29,799 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-20 00:55:29,801 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 877 msec 2023-03-20 00:55:30,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-20 00:55:30,069 INFO [Listener at localhost.localdomain/43361] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-03-20 00:55:30,079 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x458f9a9e to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:55:30,083 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6d3fb5c4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:55:30,086 DEBUG [Listener at localhost.localdomain/43361] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:55:30,090 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:37888, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:55:30,093 DEBUG [Listener at localhost.localdomain/43361] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-20 00:55:30,097 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:51744, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-20 00:55:30,105 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x20e4e4f3 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:55:30,110 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5610151a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:55:30,110 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x142c8aea to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:55:30,114 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@45864842, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:55:30,115 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x6362e5a0 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:55:30,118 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3167991b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:55:30,119 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x1f537ba4 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:55:30,122 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66ca212c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:55:30,122 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x046b5e40 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:55:30,125 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29be3c5c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:55:30,126 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x5fe050a3 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:55:30,130 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@19e3f14f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:55:30,130 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x655398b8 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:55:30,155 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@38ebb9f9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:55:30,156 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x13112432 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:55:30,170 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@76d073d8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:55:30,171 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x776686ab to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:55:30,177 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6dcc1cc1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:55:30,183 DEBUG [hconnection-0x7c9c57b5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:55:30,184 DEBUG [hconnection-0x46d9db0a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:55:30,191 DEBUG [hconnection-0x2ba8f26a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:55:30,192 DEBUG [hconnection-0x465ed1fa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:55:30,192 DEBUG [hconnection-0x34432ac-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:55:30,192 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:37896, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:55:30,195 DEBUG [hconnection-0x30a8b828-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:55:30,196 DEBUG [hconnection-0x6ad481a8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:55:30,196 DEBUG [hconnection-0x3becde13-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:55:30,198 DEBUG [hconnection-0x2f77589-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:55:30,199 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:37930, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:55:30,199 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:37902, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:55:30,200 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:37890, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:55:30,202 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:37918, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:55:30,207 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:37938, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:55:30,212 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:37946, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:55:30,212 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:37956, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:55:30,217 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:37962, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:55:30,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:30,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:30,242 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:30,242 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:30,243 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:30,243 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:30,249 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:30,250 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:30,251 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:30,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:30,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:30,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:30,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:30,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:30,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:30,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:30,254 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:30,254 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:30,254 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:30,255 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:30,260 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:30,260 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:30,262 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:30,267 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:30,267 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:30,268 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:30,269 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:30,274 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bfc994bd080bb1b8cbc4aef89c8cc98 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-20 00:55:30,294 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=A 2023-03-20 00:55:30,295 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:30,296 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=B 2023-03-20 00:55:30,296 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:30,296 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=C 2023-03-20 00:55:30,296 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:30,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:30,304 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:30,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:30,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37902 deadline: 1679273790395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:30,442 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679273730370, max timestamp=1679273730391Num uniques 859; 2023-03-20 00:55:30,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-20 00:55:30,442 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679273730370, max timestamp=1679273730391Num uniques 859; 2023-03-20 00:55:30,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 00:55:30,442 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679273730370, max timestamp=1679273730394Num uniques 859; 2023-03-20 00:55:30,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-20 00:55:30,536 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:30,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37946 deadline: 1679273790525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:30,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679273730517, max timestamp=1679273730530Num uniques 1009; 2023-03-20 00:55:30,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679273730517, max timestamp=1679273730529Num uniques 1018; 2023-03-20 00:55:30,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-03-20 00:55:30,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-20 00:55:30,550 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679273730517, max timestamp=1679273730530Num uniques 1009; 2023-03-20 00:55:30,550 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-03-20 00:55:30,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:30,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37918 deadline: 1679273790593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:30,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679273730561, max timestamp=1679273730593Num uniques 1027; 2023-03-20 00:55:30,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-20 00:55:30,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679273730561, max timestamp=1679273730591Num uniques 1027; 2023-03-20 00:55:30,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-20 00:55:30,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679273730561, max timestamp=1679273730591Num uniques 1027; 2023-03-20 00:55:30,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=6870, suffix heap size=134776,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=150 2023-03-20 00:55:30,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:30,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37946 deadline: 1679273790649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:30,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:30,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37956 deadline: 1679273790656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:30,661 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679273730635, max timestamp=1679273730649Num uniques 1036; 2023-03-20 00:55:30,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-20 00:55:30,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679273730635, max timestamp=1679273730649Num uniques 1009; 2023-03-20 00:55:30,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-20 00:55:30,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=165, cellSize=7550, totalHeapSize=17400, min timestamp=1679273730635, max timestamp=1679273730649Num uniques 1036; 2023-03-20 00:55:30,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47436, new segment data size=7550, suffix heap size=135712,new segment heap size=17400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=165 2023-03-20 00:55:30,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:30,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37902 deadline: 1679273790712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:30,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679273730673, max timestamp=1679273730711Num uniques 1024; 2023-03-20 00:55:30,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46904, new segment data size=8930, suffix heap size=134464,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1024, new segment cells count=195 2023-03-20 00:55:30,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679273730673, max timestamp=1679273730711Num uniques 1045; 2023-03-20 00:55:30,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-03-20 00:55:30,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679273730673, max timestamp=1679273730711Num uniques 1009; 2023-03-20 00:55:30,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8930, suffix heap size=132904,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=195 2023-03-20 00:55:30,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:30,777 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679273730764, max timestamp=1679273730772Num uniques 1054; 2023-03-20 00:55:30,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37956 deadline: 1679273790771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:30,777 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-20 00:55:30,777 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679273730764, max timestamp=1679273730772Num uniques 1054; 2023-03-20 00:55:30,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-20 00:55:30,781 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679273730764, max timestamp=1679273730772Num uniques 1054; 2023-03-20 00:55:30,781 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-20 00:55:30,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=363, cellSize=16618, totalHeapSize=37992, min timestamp=1679273730783, max timestamp=1679273730813Num uniques 1013; 2023-03-20 00:55:30,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=16618, suffix heap size=133320,new segment heap size=37992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=363 2023-03-20 00:55:30,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=363, cellSize=16618, totalHeapSize=37992, min timestamp=1679273730783, max timestamp=1679273730813Num uniques 1013; 2023-03-20 00:55:30,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=16618, suffix heap size=133320,new segment heap size=37992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=363 2023-03-20 00:55:30,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=363, cellSize=16618, totalHeapSize=37992, min timestamp=1679273730783, max timestamp=1679273730813Num uniques 1013; 2023-03-20 00:55:30,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=16618, suffix heap size=133320,new segment heap size=37992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=363 2023-03-20 00:55:30,856 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:30,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37946 deadline: 1679273790850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:30,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:30,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37938 deadline: 1679273790850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:30,858 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:30,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37918 deadline: 1679273790853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:30,860 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:30,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37902 deadline: 1679273790856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:30,882 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/eea2a04a552a4392935fc730d34dde0b 2023-03-20 00:55:30,887 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:30,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37956 deadline: 1679273790882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:30,961 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:30,961 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:30,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37938 deadline: 1679273790960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:30,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37946 deadline: 1679273790960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:30,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:30,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37918 deadline: 1679273790961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:30,968 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:30,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37902 deadline: 1679273790968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,022 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/0426b3b87122418ab15e6a2d25434e33 2023-03-20 00:55:31,092 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/f90ebde976224da8b38697ea810d80ea 2023-03-20 00:55:31,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:31,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37956 deadline: 1679273791091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,103 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/eea2a04a552a4392935fc730d34dde0b as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/eea2a04a552a4392935fc730d34dde0b 2023-03-20 00:55:31,117 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/eea2a04a552a4392935fc730d34dde0b, entries=150, sequenceid=10, filesize=11.6 K 2023-03-20 00:55:31,121 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-03-20 00:55:31,124 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/0426b3b87122418ab15e6a2d25434e33 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/0426b3b87122418ab15e6a2d25434e33 2023-03-20 00:55:31,148 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/0426b3b87122418ab15e6a2d25434e33, entries=150, sequenceid=10, filesize=11.6 K 2023-03-20 00:55:31,156 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/f90ebde976224da8b38697ea810d80ea as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/f90ebde976224da8b38697ea810d80ea 2023-03-20 00:55:31,173 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/f90ebde976224da8b38697ea810d80ea, entries=150, sequenceid=10, filesize=11.6 K 2023-03-20 00:55:31,179 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=147.60 KB/151140 for 1bfc994bd080bb1b8cbc4aef89c8cc98 in 902ms, sequenceid=10, compaction requested=false 2023-03-20 00:55:31,181 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-20 00:55:31,185 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:31,185 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:31,186 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:31,186 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:31,187 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,187 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:31,187 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:31,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:31,188 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1bfc994bd080bb1b8cbc4aef89c8cc98 3/3 column families, dataSize=154.31 KB heapSize=425.63 KB 2023-03-20 00:55:31,189 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=A 2023-03-20 00:55:31,190 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:31,190 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:31,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:31,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:31,190 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=B 2023-03-20 00:55:31,192 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:31,191 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,192 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=C 2023-03-20 00:55:31,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:31,192 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:31,192 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,193 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:31,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:31,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:31,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:31,197 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:31,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e9d0fc4[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:31,197 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,203 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,204 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:31,204 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:31,204 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:31,204 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,204 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:31,211 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:31,211 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:31,211 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:31,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:31,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37918 deadline: 1679273791199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:31,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37902 deadline: 1679273791200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:31,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37946 deadline: 1679273791204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,218 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:31,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:31,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37938 deadline: 1679273791204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:31,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:31,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:31,231 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:31,231 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:31,231 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:31,231 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:31,231 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:31,240 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,241 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:31,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:31,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,241 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:31,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:31,245 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/8d333f99151a43c2901b7c4665b8ea42 2023-03-20 00:55:31,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,246 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:31,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:31,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:31,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,249 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:31,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:31,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,263 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,263 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:31,264 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:31,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:31,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:31,269 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:31,272 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,272 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,272 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,272 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,265 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:31,272 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:31,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:31,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:31,272 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:31,273 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:31,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:31,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:31,309 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/0285bbe78f1142dd88d926aa91c86070 2023-03-20 00:55:31,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:31,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37918 deadline: 1679273791324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:31,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37902 deadline: 1679273791327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:31,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37946 deadline: 1679273791328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,330 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:31,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37938 deadline: 1679273791328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,373 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:31,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:31,382 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/ac6f26105a1f46caad27266964fd5b20 2023-03-20 00:55:31,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:31,396 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:31,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37956 deadline: 1679273791396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,397 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/8d333f99151a43c2901b7c4665b8ea42 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/8d333f99151a43c2901b7c4665b8ea42 2023-03-20 00:55:31,414 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/8d333f99151a43c2901b7c4665b8ea42, entries=150, sequenceid=150, filesize=11.8 K 2023-03-20 00:55:31,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:31,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:31,415 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:31,415 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:31,418 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:31,418 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:31,419 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,419 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:31,422 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:31,422 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:31,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:31,422 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,422 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:31,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:31,422 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/0285bbe78f1142dd88d926aa91c86070 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/0285bbe78f1142dd88d926aa91c86070 2023-03-20 00:55:31,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:31,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:31,424 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:31,426 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:31,426 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:31,426 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:31,426 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:31,430 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:31,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:31,439 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:31,439 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:31,441 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:31,441 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:31,441 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:31,442 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:31,443 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:31,443 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,443 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:31,443 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:31,445 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:31,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:31,446 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:31,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:31,447 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,447 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:31,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:31,448 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/0285bbe78f1142dd88d926aa91c86070, entries=150, sequenceid=150, filesize=11.8 K 2023-03-20 00:55:31,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:31,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:31,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c3c378b[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:31,449 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:31,449 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,449 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/ac6f26105a1f46caad27266964fd5b20 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/ac6f26105a1f46caad27266964fd5b20 2023-03-20 00:55:31,450 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,450 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:31,450 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:31,450 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,450 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:31,450 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:31,455 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:31,455 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,455 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:31,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:31,455 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:31,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:31,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:31,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:31,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:31,463 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/ac6f26105a1f46caad27266964fd5b20, entries=150, sequenceid=150, filesize=11.8 K 2023-03-20 00:55:31,465 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~445.50 KB/456192, currentSize=26.84 KB/27480 for 1bfc994bd080bb1b8cbc4aef89c8cc98 in 278ms, sequenceid=150, compaction requested=false 2023-03-20 00:55:31,466 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:31,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:31,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:31,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:31,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:31,469 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:31,469 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:31,469 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:31,469 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:31,469 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:31,472 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,472 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:31,472 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:31,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,473 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:31,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:31,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:31,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:31,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:31,475 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:31,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:31,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:31,502 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:31,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:31,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:31,503 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:31,503 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:31,503 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:31,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:31,503 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:31,503 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:31,503 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,503 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,503 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,503 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,504 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:31,504 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:31,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:31,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:31,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:31,539 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1bfc994bd080bb1b8cbc4aef89c8cc98 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-20 00:55:31,540 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=A 2023-03-20 00:55:31,540 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:31,540 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=B 2023-03-20 00:55:31,540 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:31,540 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=C 2023-03-20 00:55:31,540 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:31,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:31,583 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/50f1ef0abcf3450c82a0303f781fa7f8 2023-03-20 00:55:31,604 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:31,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679273731585, max timestamp=1679273731599Num uniques 859; 2023-03-20 00:55:31,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679273731585, max timestamp=1679273731599Num uniques 859; 2023-03-20 00:55:31,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-20 00:55:31,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679273731585, max timestamp=1679273731599Num uniques 859; 2023-03-20 00:55:31,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-20 00:55:31,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-20 00:55:31,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:31,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:31,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:31,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:31,631 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:31,631 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:31,631 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:31,631 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:31,632 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,632 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:31,636 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,636 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:31,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:31,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:31,637 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,637 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:31,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:31,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:31,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:31,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:31,640 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:31,641 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:31,641 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:31,641 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:31,645 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:31,645 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:31,647 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:31,652 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:31,652 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:31,652 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:31,652 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:31,654 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:31,654 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:31,654 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,654 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:31,654 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:31,663 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,663 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:31,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:31,664 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,664 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:31,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:31,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:31,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:31,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:31,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:31,666 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:31,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20c8844[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:31,666 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,667 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,668 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:31,668 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:31,668 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:31,668 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,668 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:31,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:31,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37946 deadline: 1679273791652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,673 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:31,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:31,674 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:31,674 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:31,674 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:31,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:31,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:31,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37918 deadline: 1679273791656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:31,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:31,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37902 deadline: 1679273791660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:31,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37938 deadline: 1679273791663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:31,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:31,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:31,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:31,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:31,680 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:31,680 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:31,680 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:31,680 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:31,681 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:31,686 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,687 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:31,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,687 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:31,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:31,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:31,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:31,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:31,688 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:31,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:31,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:31,689 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:31,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:31,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,699 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:31,699 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:31,699 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:31,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:31,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:31,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:31,699 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:31,700 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:31,700 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,700 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:31,699 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:31,700 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:31,700 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,700 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,700 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:31,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:31,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:31,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:31,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37946 deadline: 1679273791778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,781 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:31,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37902 deadline: 1679273791779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,781 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:31,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37938 deadline: 1679273791779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,788 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:31,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37918 deadline: 1679273791786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,800 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:31,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:31,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:31,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:31,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:31,817 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:31,817 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:31,818 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:31,818 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:31,819 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,819 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:31,820 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,820 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:31,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:31,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:31,821 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,821 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:31,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:31,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:31,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:31,822 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:31,826 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:31,826 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:31,826 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:31,826 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:31,828 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:31,828 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:31,828 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:31,829 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:31,829 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:31,829 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:31,829 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:31,830 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:31,830 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:31,830 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,830 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:31,830 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:31,834 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,834 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:31,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:31,835 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,835 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:31,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:31,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:31,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:31,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:31,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:31,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39d81d82[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:31,838 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:31,839 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,845 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,845 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:31,845 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:31,845 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:31,856 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:31,856 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,856 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:31,857 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:31,858 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:31,858 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:31,858 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:31,858 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:31,858 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:31,858 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:31,858 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:31,867 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,867 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:31,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:31,867 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:31,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:31,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:31,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:31,870 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:31,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:31,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:31,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,872 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:31,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:31,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,877 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,877 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:31,877 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:31,877 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,877 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:31,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:31,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:31,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:31,877 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:31,878 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:31,877 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,878 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:31,878 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:31,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:31,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:31,879 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:31,879 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:31,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:31,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:31,906 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:31,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37956 deadline: 1679273791902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,979 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:31,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:31,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:31,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:31,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37946 deadline: 1679273791983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:31,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:31,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37938 deadline: 1679273791984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37902 deadline: 1679273791986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,992 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:31,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37918 deadline: 1679273791991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:31,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:31,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:31,998 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:31,998 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:32,002 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:32,002 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:32,003 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:32,003 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:32,004 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,004 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:32,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:32,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:32,004 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,004 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:32,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:32,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:32,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:32,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:32,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:32,006 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:32,006 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:32,006 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:32,006 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:32,007 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:32,010 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:32,010 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:32,010 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:32,010 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:32,010 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:32,011 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:32,011 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:32,012 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:32,012 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,012 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:32,012 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:32,020 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:32,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:32,020 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:32,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:32,021 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:32,021 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:32,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:32,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:32,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:32,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:32,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f6dc652[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:32,023 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:32,023 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:32,026 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:32,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:32,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:32,026 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:32,026 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:32,026 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:32,027 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,027 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:32,028 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:32,028 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:32,028 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,028 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:32,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:32,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:32,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:32,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:32,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:32,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:32,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:32,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:32,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:32,032 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:32,032 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:32,032 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:32,032 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:32,032 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:32,034 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:32,034 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:32,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:32,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:32,034 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:32,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:32,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:32,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:32,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:32,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:32,035 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:32,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:32,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:32,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:32,039 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:32,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:32,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:32,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,046 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,046 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:32,046 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:32,046 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,046 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:32,046 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:32,046 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:32,046 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:32,046 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:32,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:32,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:32,046 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:32,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:32,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:32,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:32,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:32,048 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:32,048 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:32,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:32,062 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/6b4c59e3f0e843e1b3a1308f31a923b7 2023-03-20 00:55:32,112 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/ba4ce765668d4051b8a1b62f3608f8f6 2023-03-20 00:55:32,123 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/50f1ef0abcf3450c82a0303f781fa7f8 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/50f1ef0abcf3450c82a0303f781fa7f8 2023-03-20 00:55:32,137 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/50f1ef0abcf3450c82a0303f781fa7f8, entries=150, sequenceid=160, filesize=11.8 K 2023-03-20 00:55:32,145 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/6b4c59e3f0e843e1b3a1308f31a923b7 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/6b4c59e3f0e843e1b3a1308f31a923b7 2023-03-20 00:55:32,160 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:32,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:32,161 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/6b4c59e3f0e843e1b3a1308f31a923b7, entries=150, sequenceid=160, filesize=11.8 K 2023-03-20 00:55:32,169 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/ba4ce765668d4051b8a1b62f3608f8f6 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/ba4ce765668d4051b8a1b62f3608f8f6 2023-03-20 00:55:32,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:32,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:32,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:32,182 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:32,182 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:32,186 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:32,186 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:32,187 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:32,187 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:32,188 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,188 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:32,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:32,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:32,188 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,188 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:32,188 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/ba4ce765668d4051b8a1b62f3608f8f6, entries=150, sequenceid=160, filesize=11.8 K 2023-03-20 00:55:32,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:32,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:32,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:32,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:32,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:32,190 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 1bfc994bd080bb1b8cbc4aef89c8cc98 in 651ms, sequenceid=160, compaction requested=true 2023-03-20 00:55:32,190 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:32,190 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:32,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:32,191 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:32,191 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:32,192 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:32,195 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:32,195 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:32,195 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:32,195 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:32,195 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bfc994bd080bb1b8cbc4aef89c8cc98 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-03-20 00:55:32,196 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=A 2023-03-20 00:55:32,196 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:32,196 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=B 2023-03-20 00:55:32,196 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:32,196 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=C 2023-03-20 00:55:32,196 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:32,199 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:55:32,199 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:55:32,199 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:55:32,199 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:55:32,206 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35958 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:55:32,209 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-03-20 00:55:32,209 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 1bfc994bd080bb1b8cbc4aef89c8cc98/B is initiating minor compaction (all files) 2023-03-20 00:55:32,209 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bfc994bd080bb1b8cbc4aef89c8cc98/B in TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:32,209 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/0426b3b87122418ab15e6a2d25434e33, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/0285bbe78f1142dd88d926aa91c86070, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/6b4c59e3f0e843e1b3a1308f31a923b7] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp, totalSize=35.1 K 2023-03-20 00:55:32,210 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-03-20 00:55:32,211 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 0426b3b87122418ab15e6a2d25434e33, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1679273730231 2023-03-20 00:55:32,214 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:55:32,218 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 0285bbe78f1142dd88d926aa91c86070, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1679273730847 2023-03-20 00:55:32,219 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 6b4c59e3f0e843e1b3a1308f31a923b7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1679273731533 2023-03-20 00:55:32,220 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35958 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:55:32,221 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 1bfc994bd080bb1b8cbc4aef89c8cc98/A is initiating minor compaction (all files) 2023-03-20 00:55:32,221 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bfc994bd080bb1b8cbc4aef89c8cc98/A in TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:32,221 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/eea2a04a552a4392935fc730d34dde0b, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/8d333f99151a43c2901b7c4665b8ea42, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/50f1ef0abcf3450c82a0303f781fa7f8] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp, totalSize=35.1 K 2023-03-20 00:55:32,222 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting eea2a04a552a4392935fc730d34dde0b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1679273730231 2023-03-20 00:55:32,223 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 8d333f99151a43c2901b7c4665b8ea42, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1679273730847 2023-03-20 00:55:32,228 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 50f1ef0abcf3450c82a0303f781fa7f8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1679273731533 2023-03-20 00:55:32,281 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1bfc994bd080bb1b8cbc4aef89c8cc98#A#compaction#11 average throughput is 0.34 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:55:32,281 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 1bfc994bd080bb1b8cbc4aef89c8cc98#B#compaction#10 average throughput is 0.39 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:55:32,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:32,314 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:32,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:32,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37918 deadline: 1679273792331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,338 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/347e198bc2284d06916b6a674d49c885 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/347e198bc2284d06916b6a674d49c885 2023-03-20 00:55:32,342 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:32,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37902 deadline: 1679273792335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:32,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37938 deadline: 1679273792335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:32,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37946 deadline: 1679273792335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,371 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1bfc994bd080bb1b8cbc4aef89c8cc98/B of 1bfc994bd080bb1b8cbc4aef89c8cc98 into 347e198bc2284d06916b6a674d49c885(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:55:32,373 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:32,374 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98., storeName=1bfc994bd080bb1b8cbc4aef89c8cc98/B, priority=13, startTime=1679273732199; duration=0sec 2023-03-20 00:55:32,376 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:55:32,376 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:55:32,378 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35958 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:55:32,378 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 1bfc994bd080bb1b8cbc4aef89c8cc98/C is initiating minor compaction (all files) 2023-03-20 00:55:32,378 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bfc994bd080bb1b8cbc4aef89c8cc98/C in TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:32,378 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/f90ebde976224da8b38697ea810d80ea, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/ac6f26105a1f46caad27266964fd5b20, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/ba4ce765668d4051b8a1b62f3608f8f6] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp, totalSize=35.1 K 2023-03-20 00:55:32,379 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting f90ebde976224da8b38697ea810d80ea, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1679273730231 2023-03-20 00:55:32,380 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting ac6f26105a1f46caad27266964fd5b20, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1679273730847 2023-03-20 00:55:32,381 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting ba4ce765668d4051b8a1b62f3608f8f6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1679273731533 2023-03-20 00:55:32,400 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 1bfc994bd080bb1b8cbc4aef89c8cc98#C#compaction#12 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:55:32,456 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/ce98b1cbce9a44878321d2521abea0d2 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/ce98b1cbce9a44878321d2521abea0d2 2023-03-20 00:55:32,457 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:32,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:32,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37938 deadline: 1679273792450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,462 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:32,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37918 deadline: 1679273792452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:32,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37946 deadline: 1679273792452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37902 deadline: 1679273792452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,482 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1bfc994bd080bb1b8cbc4aef89c8cc98/C of 1bfc994bd080bb1b8cbc4aef89c8cc98 into ce98b1cbce9a44878321d2521abea0d2(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:55:32,482 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:32,482 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98., storeName=1bfc994bd080bb1b8cbc4aef89c8cc98/C, priority=13, startTime=1679273732199; duration=0sec 2023-03-20 00:55:32,483 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:55:32,668 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:32,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37938 deadline: 1679273792665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:32,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:32,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37918 deadline: 1679273792666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37946 deadline: 1679273792666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:32,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37902 deadline: 1679273792667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,683 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/0b55f9469a7343cfbfa8e0b89fcc37f9 2023-03-20 00:55:32,727 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/fe63a3a3ddd042099d41fb6d367bc674 2023-03-20 00:55:32,741 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/2cfe9b13638244fe80adc14cfc7c5e53 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/2cfe9b13638244fe80adc14cfc7c5e53 2023-03-20 00:55:32,765 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1bfc994bd080bb1b8cbc4aef89c8cc98/A of 1bfc994bd080bb1b8cbc4aef89c8cc98 into 2cfe9b13638244fe80adc14cfc7c5e53(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:55:32,765 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:32,765 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98., storeName=1bfc994bd080bb1b8cbc4aef89c8cc98/A, priority=13, startTime=1679273732196; duration=0sec 2023-03-20 00:55:32,765 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:55:32,788 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/65630d250fff487ca88bbf272c53662e 2023-03-20 00:55:32,806 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/0b55f9469a7343cfbfa8e0b89fcc37f9 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/0b55f9469a7343cfbfa8e0b89fcc37f9 2023-03-20 00:55:32,817 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/0b55f9469a7343cfbfa8e0b89fcc37f9, entries=150, sequenceid=195, filesize=11.8 K 2023-03-20 00:55:32,819 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/fe63a3a3ddd042099d41fb6d367bc674 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/fe63a3a3ddd042099d41fb6d367bc674 2023-03-20 00:55:32,840 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/fe63a3a3ddd042099d41fb6d367bc674, entries=150, sequenceid=195, filesize=11.8 K 2023-03-20 00:55:32,844 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/65630d250fff487ca88bbf272c53662e as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/65630d250fff487ca88bbf272c53662e 2023-03-20 00:55:32,870 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/65630d250fff487ca88bbf272c53662e, entries=150, sequenceid=195, filesize=11.8 K 2023-03-20 00:55:32,873 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 1bfc994bd080bb1b8cbc4aef89c8cc98 in 678ms, sequenceid=195, compaction requested=false 2023-03-20 00:55:32,873 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:32,873 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:32,874 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:32,874 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:32,875 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,875 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:32,875 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:32,876 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:32,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:32,877 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:32,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:32,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:32,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:32,879 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:32,879 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:32,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:32,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:32,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@200345b[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:32,881 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:32,881 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:32,882 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:32,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:32,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:32,882 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:32,882 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:32,882 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:32,883 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,884 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:32,885 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,885 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:32,885 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:32,885 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:32,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:32,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:32,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:32,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:32,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:32,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:32,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:32,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:32,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:32,891 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:32,891 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:32,891 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:32,891 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:32,891 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:32,901 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:32,901 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:32,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:32,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:32,901 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:32,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:32,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:32,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:32,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:32,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:32,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:32,903 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:32,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:32,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:32,907 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:32,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:32,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:32,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:32,914 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1bfc994bd080bb1b8cbc4aef89c8cc98 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-20 00:55:32,914 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=A 2023-03-20 00:55:32,915 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:32,915 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=B 2023-03-20 00:55:32,915 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:32,915 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=C 2023-03-20 00:55:32,915 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:32,917 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:32,918 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:32,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:32,918 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:32,918 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:32,918 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:32,918 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:32,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:32,920 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:32,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:32,920 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:32,920 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:32,920 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:32,920 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:32,921 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:32,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:32,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:32,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:32,960 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/b1fd3ce5b70a458cb9e00a3c3d5e8d26 2023-03-20 00:55:32,974 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-20 00:55:33,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679273732976, max timestamp=1679273732997Num uniques 859; 2023-03-20 00:55:33,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-20 00:55:33,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679273732976, max timestamp=1679273732997Num uniques 859; 2023-03-20 00:55:33,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-20 00:55:33,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679273732976, max timestamp=1679273732997Num uniques 859; 2023-03-20 00:55:33,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 00:55:33,021 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:33,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:33,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:33,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:33,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37946 deadline: 1679273793033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:33,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:33,051 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:33,051 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:33,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:33,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37902 deadline: 1679273793039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,066 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:33,067 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:33,067 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:33,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37956 deadline: 1679273793057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,071 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:33,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37918 deadline: 1679273793067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,068 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,072 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:33,075 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:33,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:33,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:33,075 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:33,076 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,076 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:33,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:33,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:33,077 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:33,078 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:33,078 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:33,078 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:33,078 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:33,079 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:33,079 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:33,079 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:33,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:33,080 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:33,080 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:33,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37938 deadline: 1679273793073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,080 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:33,080 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:33,080 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:33,080 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:33,080 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,080 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:33,080 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:33,082 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,082 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:33,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:33,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:33,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:33,083 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,083 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:33,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:33,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:33,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:33,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6428d2f7[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:33,085 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:33,085 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,086 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,086 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:33,086 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:33,087 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:33,088 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:33,088 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:33,088 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:33,088 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,089 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:33,089 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:33,089 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:33,089 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:33,089 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:33,089 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:33,089 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:33,094 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,094 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:33,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:33,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:33,095 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:33,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:33,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:33,097 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:33,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:33,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,100 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:33,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:33,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,104 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:33,104 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:33,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:33,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:33,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:33,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:33,106 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,106 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:33,106 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:33,106 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,106 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:33,106 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:33,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:33,106 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,108 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,109 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:33,109 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:33,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:33,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37946 deadline: 1679273793160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:33,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37902 deadline: 1679273793161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:33,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37956 deadline: 1679273793173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:33,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37918 deadline: 1679273793173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,184 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:33,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37938 deadline: 1679273793182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,209 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:33,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:33,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:33,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:33,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:33,214 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:33,214 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:33,215 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:33,215 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:33,215 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,215 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:33,216 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:33,216 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:33,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:33,217 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,217 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:33,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:33,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:33,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:33,218 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:33,219 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:33,219 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:33,219 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:33,219 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:33,220 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:33,223 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:33,223 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:33,226 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:33,226 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:33,226 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:33,226 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:33,226 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:33,226 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:33,227 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,227 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:33,227 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:33,237 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,237 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:33,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:33,238 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,238 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:33,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:33,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:33,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:33,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d65e7fd[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:33,240 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:33,240 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,245 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,245 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:33,245 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:33,245 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:33,245 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,245 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:33,247 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,247 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:33,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:33,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:33,247 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:33,247 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:33,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:33,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:33,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:33,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:33,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:33,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:33,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:33,256 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:33,256 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:33,256 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:33,256 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:33,256 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:33,261 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,262 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,262 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:33,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:33,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:33,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:33,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:33,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:33,263 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:33,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:33,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:33,269 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:33,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:33,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,273 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:33,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:33,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:33,273 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:33,273 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,273 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:33,273 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:33,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:33,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:33,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:33,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:33,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:33,274 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:33,274 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,274 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,274 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,274 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,274 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:33,274 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:33,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:33,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37946 deadline: 1679273793363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:33,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37902 deadline: 1679273793367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,375 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:33,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:33,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:33,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:33,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:33,387 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:33,387 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:33,387 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:33,388 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:33,388 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,389 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:33,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37956 deadline: 1679273793379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,389 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:33,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37918 deadline: 1679273793379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,389 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:33,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:33,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37938 deadline: 1679273793389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,391 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:33,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:33,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:33,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:33,391 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,392 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,392 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:33,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:33,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:33,392 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:33,394 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:33,394 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:33,394 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:33,394 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:33,395 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:33,396 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:33,396 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:33,396 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:33,396 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:33,396 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:33,396 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:33,396 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:33,396 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:33,396 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,396 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:33,396 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:33,398 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,398 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:33,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:33,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:33,399 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:33,399 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:33,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:33,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cbbed55[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:33,402 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:33,402 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,403 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,404 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,404 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:33,404 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:33,404 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:33,404 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:33,405 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:33,405 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:33,405 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:33,407 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:33,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:33,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:33,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:33,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:33,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:33,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:33,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:33,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:33,413 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:33,413 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:33,414 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:33,414 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:33,414 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:33,428 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,428 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,428 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:33,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:33,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:33,429 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:33,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:33,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:33,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:33,433 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,437 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:33,437 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,437 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:33,437 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:33,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:33,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:33,437 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:33,438 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,438 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,438 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,438 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,438 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:33,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:33,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:33,439 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:33,439 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:33,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:33,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:33,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:33,477 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/aeab3abfa1874b35bc3239c7e015c4bd 2023-03-20 00:55:33,508 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/15525c7551e2466d81d50c5cb82bd03c 2023-03-20 00:55:33,521 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/b1fd3ce5b70a458cb9e00a3c3d5e8d26 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/b1fd3ce5b70a458cb9e00a3c3d5e8d26 2023-03-20 00:55:33,532 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/b1fd3ce5b70a458cb9e00a3c3d5e8d26, entries=150, sequenceid=209, filesize=11.8 K 2023-03-20 00:55:33,537 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/aeab3abfa1874b35bc3239c7e015c4bd as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/aeab3abfa1874b35bc3239c7e015c4bd 2023-03-20 00:55:33,539 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:33,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:33,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:33,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:33,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:33,546 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:33,546 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:33,547 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:33,547 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:33,548 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,548 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:33,549 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,549 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:33,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:33,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:33,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:33,549 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,550 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:33,550 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/aeab3abfa1874b35bc3239c7e015c4bd, entries=150, sequenceid=209, filesize=11.8 K 2023-03-20 00:55:33,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:33,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:33,550 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:33,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:33,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:33,554 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:33,554 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:33,556 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:33,556 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/15525c7551e2466d81d50c5cb82bd03c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/15525c7551e2466d81d50c5cb82bd03c 2023-03-20 00:55:33,556 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:33,556 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:33,556 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:33,558 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:33,559 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:33,559 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:33,559 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:33,559 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:33,559 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,559 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:33,559 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:33,566 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/15525c7551e2466d81d50c5cb82bd03c, entries=150, sequenceid=209, filesize=11.8 K 2023-03-20 00:55:33,569 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,569 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:33,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:33,570 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,570 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:33,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:33,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:33,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:33,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:33,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:33,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f1ab075[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:33,572 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:33,572 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,576 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,576 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:33,577 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,577 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:33,577 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:33,577 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:33,578 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:33,578 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,578 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=143.13 KB/146560 for 1bfc994bd080bb1b8cbc4aef89c8cc98 in 664ms, sequenceid=209, compaction requested=true 2023-03-20 00:55:33,578 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:33,578 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:33,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:33,578 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:33,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:33,581 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:55:33,581 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:55:33,581 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:55:33,581 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:55:33,581 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:55:33,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:33,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:33,583 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36211 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:55:33,583 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36211 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:55:33,583 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 1bfc994bd080bb1b8cbc4aef89c8cc98/A is initiating minor compaction (all files) 2023-03-20 00:55:33,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:33,583 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 1bfc994bd080bb1b8cbc4aef89c8cc98/B is initiating minor compaction (all files) 2023-03-20 00:55:33,583 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bfc994bd080bb1b8cbc4aef89c8cc98/A in TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:33,583 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bfc994bd080bb1b8cbc4aef89c8cc98/B in TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:33,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,583 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/347e198bc2284d06916b6a674d49c885, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/fe63a3a3ddd042099d41fb6d367bc674, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/aeab3abfa1874b35bc3239c7e015c4bd] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp, totalSize=35.4 K 2023-03-20 00:55:33,583 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/2cfe9b13638244fe80adc14cfc7c5e53, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/0b55f9469a7343cfbfa8e0b89fcc37f9, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/b1fd3ce5b70a458cb9e00a3c3d5e8d26] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp, totalSize=35.4 K 2023-03-20 00:55:33,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:33,584 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 347e198bc2284d06916b6a674d49c885, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1679273731533 2023-03-20 00:55:33,585 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 2cfe9b13638244fe80adc14cfc7c5e53, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1679273731533 2023-03-20 00:55:33,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:33,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,586 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 0b55f9469a7343cfbfa8e0b89fcc37f9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1679273731607 2023-03-20 00:55:33,586 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting fe63a3a3ddd042099d41fb6d367bc674, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1679273731607 2023-03-20 00:55:33,587 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting b1fd3ce5b70a458cb9e00a3c3d5e8d26, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679273732332 2023-03-20 00:55:33,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:33,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:33,587 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:33,587 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:33,587 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:33,587 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:33,587 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:33,587 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting aeab3abfa1874b35bc3239c7e015c4bd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679273732332 2023-03-20 00:55:33,593 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,593 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:33,593 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:33,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:33,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:33,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:33,594 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:33,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:33,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,599 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:33,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,609 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,609 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:33,609 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:33,609 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,610 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,610 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:33,610 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:33,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:33,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:33,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:33,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:33,610 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:33,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:33,610 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:33,610 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,610 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:33,610 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:33,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:33,617 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1bfc994bd080bb1b8cbc4aef89c8cc98#A#compaction#18 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:55:33,622 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 1bfc994bd080bb1b8cbc4aef89c8cc98#B#compaction#19 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:55:33,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679273733026, max timestamp=1679273733674Num uniques 1118; 2023-03-20 00:55:33,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-03-20 00:55:33,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:33,679 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1bfc994bd080bb1b8cbc4aef89c8cc98 3/3 column families, dataSize=107.34 KB heapSize=302.36 KB 2023-03-20 00:55:33,679 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=A 2023-03-20 00:55:33,679 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:33,679 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=B 2023-03-20 00:55:33,679 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-20 00:55:33,679 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=C 2023-03-20 00:55:33,679 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-20 00:55:33,689 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/0146222707c242238955d67ab2f3ea96 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/0146222707c242238955d67ab2f3ea96 2023-03-20 00:55:33,703 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1bfc994bd080bb1b8cbc4aef89c8cc98/B of 1bfc994bd080bb1b8cbc4aef89c8cc98 into 0146222707c242238955d67ab2f3ea96(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:55:33,703 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:33,704 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98., storeName=1bfc994bd080bb1b8cbc4aef89c8cc98/B, priority=13, startTime=1679273733581; duration=0sec 2023-03-20 00:55:33,704 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:55:33,704 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:55:33,706 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36211 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:55:33,706 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 1bfc994bd080bb1b8cbc4aef89c8cc98/C is initiating minor compaction (all files) 2023-03-20 00:55:33,706 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bfc994bd080bb1b8cbc4aef89c8cc98/C in TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:33,707 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/ce98b1cbce9a44878321d2521abea0d2, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/65630d250fff487ca88bbf272c53662e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/15525c7551e2466d81d50c5cb82bd03c] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp, totalSize=35.4 K 2023-03-20 00:55:33,707 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting ce98b1cbce9a44878321d2521abea0d2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1679273731533 2023-03-20 00:55:33,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:33,710 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:33,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:33,709 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 65630d250fff487ca88bbf272c53662e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1679273731607 2023-03-20 00:55:33,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37918 deadline: 1679273793701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37938 deadline: 1679273793700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:33,714 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 15525c7551e2466d81d50c5cb82bd03c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679273732332 2023-03-20 00:55:33,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:33,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:33,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:33,726 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:33,726 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:33,729 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:33,729 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:33,729 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,730 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:33,730 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/3df40fd94e3042ee86071d652342991e 2023-03-20 00:55:33,730 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,730 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:33,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:33,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:33,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:33,731 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,731 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:33,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:33,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:33,732 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:33,732 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:33,732 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:33,733 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:33,733 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:33,733 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:33,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37946 deadline: 1679273793706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,734 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:33,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:33,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37902 deadline: 1679273793708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:33,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37956 deadline: 1679273793709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,751 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:33,751 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:33,754 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:33,755 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:33,755 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:33,755 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:33,755 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:33,755 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:33,755 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,757 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 1bfc994bd080bb1b8cbc4aef89c8cc98#C#compaction#21 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:55:33,763 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:33,763 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:33,764 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:33,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:33,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:33,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:33,766 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,767 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,767 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:33,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:33,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:33,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:33,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d87760b[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:33,768 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:33,768 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,781 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,787 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:33,787 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:33,787 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:33,787 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,787 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:33,788 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,788 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:33,788 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:33,788 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:33,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:33,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:33,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:33,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:33,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:33,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,802 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/03d073d5708c48b9ac8404b6a27e310e as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/03d073d5708c48b9ac8404b6a27e310e 2023-03-20 00:55:33,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:33,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:33,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:33,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:33,809 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:33,810 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:33,810 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:33,810 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:33,810 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:33,813 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,813 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:33,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:33,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:33,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,815 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/a044c4f4d79d4ebdacc14e00f6db10fb 2023-03-20 00:55:33,816 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:33,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:33,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:33,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:33,819 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:33,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:33,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:33,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:33,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37938 deadline: 1679273793817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,825 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:33,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:33,825 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1bfc994bd080bb1b8cbc4aef89c8cc98/C of 1bfc994bd080bb1b8cbc4aef89c8cc98 into 03d073d5708c48b9ac8404b6a27e310e(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:55:33,828 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:33,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37918 deadline: 1679273793817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,828 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98., storeName=1bfc994bd080bb1b8cbc4aef89c8cc98/C, priority=13, startTime=1679273733581; duration=0sec 2023-03-20 00:55:33,831 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:55:33,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,833 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,833 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:33,834 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:33,834 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:33,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:33,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:33,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:33,834 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:33,835 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,835 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,835 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:33,835 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,834 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:33,835 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:33,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:33,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:33,836 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:33,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:33,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:33,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:33,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37946 deadline: 1679273793842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,845 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:33,845 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:33,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37902 deadline: 1679273793842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37956 deadline: 1679273793842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,893 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/bccd87ae84bc4812b5092589c3ae7c21 2023-03-20 00:55:33,903 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/3df40fd94e3042ee86071d652342991e as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/3df40fd94e3042ee86071d652342991e 2023-03-20 00:55:33,912 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/3df40fd94e3042ee86071d652342991e, entries=200, sequenceid=247, filesize=14.1 K 2023-03-20 00:55:33,915 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/a044c4f4d79d4ebdacc14e00f6db10fb as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/a044c4f4d79d4ebdacc14e00f6db10fb 2023-03-20 00:55:33,925 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/a044c4f4d79d4ebdacc14e00f6db10fb, entries=150, sequenceid=247, filesize=11.8 K 2023-03-20 00:55:33,930 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/bccd87ae84bc4812b5092589c3ae7c21 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/bccd87ae84bc4812b5092589c3ae7c21 2023-03-20 00:55:33,936 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:33,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:33,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:33,943 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/bccd87ae84bc4812b5092589c3ae7c21, entries=150, sequenceid=247, filesize=11.8 K 2023-03-20 00:55:33,944 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~302.31 KB/309568, currentSize=73.80 KB/75570 for 1bfc994bd080bb1b8cbc4aef89c8cc98 in 265ms, sequenceid=247, compaction requested=false 2023-03-20 00:55:33,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:33,944 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:33,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:33,945 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:33,945 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:33,950 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:33,950 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:33,950 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,951 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:33,952 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:33,952 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:33,953 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:33,954 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:33,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:33,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:33,955 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:33,955 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:33,955 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:33,958 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:33,959 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:33,959 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:33,959 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:33,959 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:33,959 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bfc994bd080bb1b8cbc4aef89c8cc98 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-20 00:55:33,959 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=A 2023-03-20 00:55:33,959 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:33,960 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=B 2023-03-20 00:55:33,960 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:33,960 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=C 2023-03-20 00:55:33,960 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:33,991 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/5aabcf44d83140bd943c5c4bed40cf52 2023-03-20 00:55:34,029 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/34ea5ab100ef432389f26d243ad17f50 2023-03-20 00:55:34,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:34,035 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:34,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:34,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37946 deadline: 1679273794062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:34,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37956 deadline: 1679273794063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:34,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37938 deadline: 1679273794064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:34,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37918 deadline: 1679273794065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:34,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37902 deadline: 1679273794066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,101 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/a16bedf52d5949e5b9fd61eadaad3b08 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/a16bedf52d5949e5b9fd61eadaad3b08 2023-03-20 00:55:34,113 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1bfc994bd080bb1b8cbc4aef89c8cc98/A of 1bfc994bd080bb1b8cbc4aef89c8cc98 into a16bedf52d5949e5b9fd61eadaad3b08(size=12.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:55:34,113 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:34,113 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98., storeName=1bfc994bd080bb1b8cbc4aef89c8cc98/A, priority=13, startTime=1679273733581; duration=0sec 2023-03-20 00:55:34,113 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:55:34,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:34,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:34,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37946 deadline: 1679273794172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37956 deadline: 1679273794170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:34,175 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:34,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37918 deadline: 1679273794172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37938 deadline: 1679273794172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:34,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37902 deadline: 1679273794173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:34,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37946 deadline: 1679273794377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:34,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37956 deadline: 1679273794377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:34,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37918 deadline: 1679273794378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,386 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:34,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37938 deadline: 1679273794385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:34,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37902 deadline: 1679273794385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,407 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-20 00:55:34,407 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-03-20 00:55:34,493 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/93d6ccb109e846fdb81ab29f37053252 2023-03-20 00:55:34,509 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/5aabcf44d83140bd943c5c4bed40cf52 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/5aabcf44d83140bd943c5c4bed40cf52 2023-03-20 00:55:34,518 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/5aabcf44d83140bd943c5c4bed40cf52, entries=150, sequenceid=263, filesize=11.9 K 2023-03-20 00:55:34,520 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/34ea5ab100ef432389f26d243ad17f50 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/34ea5ab100ef432389f26d243ad17f50 2023-03-20 00:55:34,532 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/34ea5ab100ef432389f26d243ad17f50, entries=150, sequenceid=263, filesize=11.9 K 2023-03-20 00:55:34,537 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/93d6ccb109e846fdb81ab29f37053252 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/93d6ccb109e846fdb81ab29f37053252 2023-03-20 00:55:34,549 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/93d6ccb109e846fdb81ab29f37053252, entries=150, sequenceid=263, filesize=11.9 K 2023-03-20 00:55:34,551 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 1bfc994bd080bb1b8cbc4aef89c8cc98 in 592ms, sequenceid=263, compaction requested=true 2023-03-20 00:55:34,551 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:34,551 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:34,551 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:34,560 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:34,560 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,560 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:34,560 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:34,562 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,562 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:34,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:34,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:34,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:34,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:34,563 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:34,563 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:34,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:34,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:34,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:34,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fef3449[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:34,564 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:34,564 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:34,565 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:34,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:34,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:34,566 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,566 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:34,566 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:34,566 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:34,566 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:34,568 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:34,568 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:34,568 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:34,568 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:34,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:34,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:34,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:34,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:34,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:34,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:34,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:34,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:34,574 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:34,574 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:34,574 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:34,574 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:34,574 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:34,578 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:34,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:34,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:34,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:34,578 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:34,579 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:34,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:34,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:34,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:34,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:34,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:34,580 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:34,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:34,582 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:34,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:34,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,586 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:34,586 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:34,586 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,587 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:34,587 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:34,587 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,587 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:34,587 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:34,587 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:34,587 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:34,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:34,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:34,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:34,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:34,588 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:34,588 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:34,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:34,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:34,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:34,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:34,685 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1bfc994bd080bb1b8cbc4aef89c8cc98 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-20 00:55:34,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679273734057, max timestamp=1679273734685Num uniques 859; 2023-03-20 00:55:34,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 00:55:34,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679273734057, max timestamp=1679273734685Num uniques 859; 2023-03-20 00:55:34,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 00:55:34,688 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=A 2023-03-20 00:55:34,688 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:34,688 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=B 2023-03-20 00:55:34,688 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:34,688 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=C 2023-03-20 00:55:34,688 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:34,688 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:34,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:34,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:34,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:34,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:34,707 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:34,707 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:34,715 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:34,715 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:34,717 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:34,717 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:34,723 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:34,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:34,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:34,723 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:34,726 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,726 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:34,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:34,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:34,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:34,727 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:34,727 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:34,727 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:34,728 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:34,728 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:34,728 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:34,730 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:34,730 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:34,730 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:34,730 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:34,730 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:34,731 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:34,731 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:34,731 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:34,731 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,731 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:34,731 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:34,736 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:34,736 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:34,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:34,737 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:34,737 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:34,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:34,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:34,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:34,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:34,741 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:34,741 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:34,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37902 deadline: 1679273794732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37946 deadline: 1679273794734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679273734699, max timestamp=1679273734739Num uniques 859; 2023-03-20 00:55:34,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:34,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37938 deadline: 1679273794737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,742 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:34,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:34,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37956 deadline: 1679273794736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679273734699, max timestamp=1679273734739Num uniques 859; 2023-03-20 00:55:34,745 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:34,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c7db1a4[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:34,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679273734699, max timestamp=1679273734739Num uniques 859; 2023-03-20 00:55:34,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-20 00:55:34,746 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-20 00:55:34,746 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:34,746 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-20 00:55:34,747 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:34,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:34,747 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:34,747 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:34,747 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,747 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:34,747 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:34,749 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,749 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:34,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:34,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:34,749 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:34,749 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:34,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:34,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:34,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:34,753 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:34,753 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:34,753 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:34,753 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:34,753 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:34,761 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:34,761 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:34,761 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:34,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:34,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:34,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:34,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:34,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:34,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:34,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:34,762 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:34,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:34,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:34,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:34,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:34,764 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:34,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:34,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,769 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,769 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:34,769 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:34,770 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,770 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:34,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:34,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:34,770 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:34,770 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:34,770 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:34,771 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:34,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:34,770 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:34,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:34,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:34,772 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:34,772 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:34,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:34,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:34,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:34,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37918 deadline: 1679273794797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:34,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37902 deadline: 1679273794846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:34,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37946 deadline: 1679273794846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:34,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37938 deadline: 1679273794847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,849 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:34,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37956 deadline: 1679273794847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,872 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:34,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:34,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:34,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:34,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:34,876 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:34,876 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:34,877 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:34,877 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:34,877 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:34,878 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:34,878 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,878 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:34,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:34,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:34,879 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,879 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:34,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:34,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:34,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:34,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:34,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:34,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:34,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:34,880 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:34,880 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:34,880 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:34,880 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:34,881 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:34,881 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:34,881 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:34,881 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:34,881 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:34,881 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:34,881 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:34,881 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,881 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:34,881 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:34,882 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,882 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:34,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:34,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:34,882 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:34,882 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:34,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:34,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:34,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:34,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:34,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:34,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ae2946[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:34,887 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:34,888 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:34,894 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:34,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:34,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:34,895 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:34,895 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:34,895 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,895 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:34,895 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:34,901 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:34,901 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:34,901 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:34,901 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:34,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:34,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:34,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:34,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:34,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:34,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37918 deadline: 1679273794901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:34,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:34,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:34,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:34,905 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:34,906 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:34,906 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:34,906 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:34,906 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:34,911 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:34,911 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:34,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:34,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:34,911 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:34,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:34,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:34,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:34,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:34,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:34,912 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:34,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:34,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:34,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:34,913 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:34,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:34,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:34,917 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:34,917 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:34,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:34,917 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,918 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:34,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:34,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:34,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:34,917 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:34,918 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:34,918 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:34,918 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:34,918 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:34,919 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:34,919 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:34,919 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:34,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:34,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:34,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:35,019 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:35,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:35,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:35,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:35,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:35,023 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:35,023 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:35,024 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:35,024 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:35,025 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,025 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:35,026 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,026 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:35,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:35,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:35,026 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,026 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:35,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:35,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:35,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:35,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:35,027 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:35,027 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:35,027 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:35,027 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:35,028 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:35,028 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:35,028 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:35,028 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:35,028 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:35,028 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:35,028 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:35,028 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:35,028 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:35,028 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,029 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:35,029 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:35,030 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,030 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:35,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:35,030 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,030 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:35,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:35,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:35,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:35,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:35,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:35,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f20b6[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:35,034 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:35,034 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,035 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,035 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:35,035 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:35,036 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:35,036 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,036 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:35,036 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,036 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:35,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:35,036 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:35,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:35,037 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:35,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:35,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:35,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:35,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:35,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:35,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:35,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:35,040 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:35,040 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:35,040 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:35,040 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:35,040 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:35,041 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,041 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:35,041 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:35,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:35,042 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:35,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:35,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:35,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,045 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:35,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:35,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,048 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:35,048 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:35,048 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:35,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:35,048 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:35,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:35,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:35,048 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:35,048 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:35,048 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,048 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,048 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:35,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:35,048 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:35,048 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,049 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:35,054 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:35,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37902 deadline: 1679273795051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:35,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37946 deadline: 1679273795051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:35,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:35,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37956 deadline: 1679273795052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37938 deadline: 1679273795052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:35,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37918 deadline: 1679273795105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,125 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/29529002dd4e497e9a959a58e65929be 2023-03-20 00:55:35,146 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/b5099528c0ca480aba53d1e3c17ca99c 2023-03-20 00:55:35,149 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:35,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:35,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:35,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:35,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:35,156 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:35,156 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:35,157 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:35,157 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:35,158 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,158 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:35,159 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:35,159 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:35,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:35,159 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,159 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:35,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:35,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:35,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:35,163 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:35,172 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:35,172 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:35,172 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:35,172 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:35,173 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:35,174 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:35,180 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:35,180 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:35,180 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:35,180 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:35,181 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:35,181 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:35,182 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:35,182 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,182 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:35,182 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:35,185 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,185 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:35,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:35,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:35,186 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,186 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:35,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:35,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:35,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:35,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:35,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76a41180[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:35,188 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:35,188 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,189 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,189 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:35,189 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:35,189 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:35,189 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,189 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:35,190 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,190 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:35,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:35,190 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:35,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:35,190 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:35,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:35,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:35,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:35,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:35,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:35,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:35,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:35,193 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:35,193 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:35,193 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:35,193 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:35,193 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:35,203 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,203 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,203 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:35,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:35,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:35,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:35,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:35,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:35,204 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:35,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:35,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:35,206 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:35,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:35,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,212 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:35,212 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:35,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:35,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:35,212 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:35,212 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:35,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:35,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:35,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:35,212 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:35,213 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:35,213 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:35,214 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:35,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:35,213 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,214 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,214 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,214 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,314 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:35,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:35,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:35,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:35,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:35,318 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:35,318 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:35,319 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:35,319 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:35,320 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,320 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:35,321 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:35,321 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:35,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:35,321 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,321 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:35,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:35,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:35,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:35,322 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:35,322 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:35,322 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:35,322 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:35,323 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:35,323 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:35,324 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:35,324 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:35,324 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:35,324 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:35,324 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:35,324 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:35,324 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:35,324 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:35,324 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,324 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:35,324 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:35,325 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,325 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:35,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:35,325 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,326 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:35,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:35,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:35,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:35,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:35,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:35,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57756e61[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:35,331 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:35,331 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,336 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,336 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:35,336 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:35,337 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:35,338 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,338 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:35,342 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:35,342 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:35,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:35,343 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:35,343 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:35,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:35,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:35,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:35,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:35,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:35,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:35,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:35,346 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:35,347 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:35,347 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:35,347 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:35,347 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:35,353 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,353 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:35,353 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:35,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:35,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:35,354 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:35,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:35,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:35,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:35,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:35,355 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:35,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:35,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,361 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:35,361 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37902 deadline: 1679273795358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,361 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:35,362 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:35,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:35,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:35,362 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:35,362 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:35,362 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:35,370 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:35,370 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,370 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:35,370 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,370 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:35,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:35,370 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,370 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:35,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:35,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37938 deadline: 1679273795359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:35,371 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:35,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37956 deadline: 1679273795359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:35,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37946 deadline: 1679273795358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,414 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:35,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37918 deadline: 1679273795412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,470 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:35,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:35,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:35,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:35,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:35,475 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:35,475 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:35,476 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:35,476 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:35,476 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,476 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:35,477 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:35,477 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:35,477 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,477 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:35,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:35,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:35,478 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:35,478 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:35,478 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:35,479 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:35,479 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:35,479 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:35,482 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:35,482 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:35,482 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:35,482 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:35,482 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:35,482 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:35,482 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:35,482 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:35,482 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,482 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:35,482 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:35,484 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,484 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:35,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:35,484 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,484 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:35,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:35,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:35,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:35,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:35,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:35,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@206689d0[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:35,485 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:35,486 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,486 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,486 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:35,487 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,487 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:35,487 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:35,487 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:35,489 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,489 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:35,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:35,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:35,489 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:35,489 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:35,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:35,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:35,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:35,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:35,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:35,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:35,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:35,492 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:35,492 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:35,492 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:35,492 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:35,492 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:35,493 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,493 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:35,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:35,493 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:35,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,493 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:35,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:35,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:35,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:35,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:35,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:35,495 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:35,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:35,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,497 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:35,497 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:35,497 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:35,497 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:35,497 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:35,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:35,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:35,497 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:35,498 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,498 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,498 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,498 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,498 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:35,498 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:35,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:35,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:35,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:35,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:35,584 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/aa41caeb809640c9839f400a70eb33c1 2023-03-20 00:55:35,594 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/29529002dd4e497e9a959a58e65929be as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/29529002dd4e497e9a959a58e65929be 2023-03-20 00:55:35,598 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:35,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:35,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:35,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:35,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:35,608 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:35,608 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:35,609 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:35,609 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:35,609 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,609 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:35,610 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/29529002dd4e497e9a959a58e65929be, entries=150, sequenceid=285, filesize=11.9 K 2023-03-20 00:55:35,611 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,611 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:35,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:35,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:35,612 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,612 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:35,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:35,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,613 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/b5099528c0ca480aba53d1e3c17ca99c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/b5099528c0ca480aba53d1e3c17ca99c 2023-03-20 00:55:35,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:35,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:35,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:35,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:35,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:35,613 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:35,613 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:35,614 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:35,614 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:35,614 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:35,614 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:35,615 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:35,615 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:35,615 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:35,615 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:35,615 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:35,615 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,615 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:35,615 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:35,617 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:35,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:35,618 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:35,619 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,619 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:35,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:35,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:35,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:35,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:35,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@598507fa[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:35,623 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:35,624 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,628 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,628 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:35,628 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:35,628 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:35,628 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,628 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:35,629 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/b5099528c0ca480aba53d1e3c17ca99c, entries=150, sequenceid=285, filesize=11.9 K 2023-03-20 00:55:35,630 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:35,630 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,630 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:35,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:35,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:35,630 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:35,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:35,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:35,635 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/aa41caeb809640c9839f400a70eb33c1 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/aa41caeb809640c9839f400a70eb33c1 2023-03-20 00:55:35,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:35,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:35,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:35,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:35,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:35,637 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:35,637 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:35,637 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:35,637 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:35,637 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:35,642 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/aa41caeb809640c9839f400a70eb33c1, entries=150, sequenceid=285, filesize=11.9 K 2023-03-20 00:55:35,644 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~58.14 KB/59540, heapSize ~164.13 KB/168064, currentSize=134.18 KB/137400 for 1bfc994bd080bb1b8cbc4aef89c8cc98 in 959ms, sequenceid=285, compaction requested=true 2023-03-20 00:55:35,644 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:35,644 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:55:35,644 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 00:55:35,644 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 00:55:35,645 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,644 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:55:35,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:35,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:35,644 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,645 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:55:35,645 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:35,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:35,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:35,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:35,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,647 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50990 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 00:55:35,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:35,647 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,647 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48600 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 00:55:35,647 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 1bfc994bd080bb1b8cbc4aef89c8cc98/B is initiating minor compaction (all files) 2023-03-20 00:55:35,647 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 1bfc994bd080bb1b8cbc4aef89c8cc98/A is initiating minor compaction (all files) 2023-03-20 00:55:35,647 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bfc994bd080bb1b8cbc4aef89c8cc98/B in TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:35,647 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bfc994bd080bb1b8cbc4aef89c8cc98/A in TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:35,648 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/0146222707c242238955d67ab2f3ea96, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/a044c4f4d79d4ebdacc14e00f6db10fb, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/34ea5ab100ef432389f26d243ad17f50, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/b5099528c0ca480aba53d1e3c17ca99c] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp, totalSize=47.5 K 2023-03-20 00:55:35,648 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/a16bedf52d5949e5b9fd61eadaad3b08, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/3df40fd94e3042ee86071d652342991e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/5aabcf44d83140bd943c5c4bed40cf52, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/29529002dd4e497e9a959a58e65929be] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp, totalSize=49.8 K 2023-03-20 00:55:35,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,649 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,650 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 0146222707c242238955d67ab2f3ea96, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679273732332 2023-03-20 00:55:35,651 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting a16bedf52d5949e5b9fd61eadaad3b08, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679273732332 2023-03-20 00:55:35,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:35,651 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting a044c4f4d79d4ebdacc14e00f6db10fb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1679273733032 2023-03-20 00:55:35,652 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 3df40fd94e3042ee86071d652342991e, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1679273733026 2023-03-20 00:55:35,652 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 5aabcf44d83140bd943c5c4bed40cf52, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1679273733696 2023-03-20 00:55:35,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:35,653 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 29529002dd4e497e9a959a58e65929be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1679273734064 2023-03-20 00:55:35,653 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 34ea5ab100ef432389f26d243ad17f50, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1679273733696 2023-03-20 00:55:35,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,654 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting b5099528c0ca480aba53d1e3c17ca99c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1679273734064 2023-03-20 00:55:35,659 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:35,659 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:35,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:35,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:35,659 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:35,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:35,659 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:35,659 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:35,660 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:35,660 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:35,663 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:35,663 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:35,663 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,663 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:35,663 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:35,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:35,684 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 1bfc994bd080bb1b8cbc4aef89c8cc98#B#compaction#30 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:55:35,685 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1bfc994bd080bb1b8cbc4aef89c8cc98#A#compaction#31 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:55:35,712 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/a1a08a26289243bbaa20ca0fecaf3ed8 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/a1a08a26289243bbaa20ca0fecaf3ed8 2023-03-20 00:55:35,720 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1bfc994bd080bb1b8cbc4aef89c8cc98/A of 1bfc994bd080bb1b8cbc4aef89c8cc98 into a1a08a26289243bbaa20ca0fecaf3ed8(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:55:35,720 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:35,721 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98., storeName=1bfc994bd080bb1b8cbc4aef89c8cc98/A, priority=12, startTime=1679273735644; duration=0sec 2023-03-20 00:55:35,721 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:55:35,721 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 00:55:35,724 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48600 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 00:55:35,724 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 1bfc994bd080bb1b8cbc4aef89c8cc98/C is initiating minor compaction (all files) 2023-03-20 00:55:35,724 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bfc994bd080bb1b8cbc4aef89c8cc98/C in TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:35,724 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/03d073d5708c48b9ac8404b6a27e310e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/bccd87ae84bc4812b5092589c3ae7c21, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/93d6ccb109e846fdb81ab29f37053252, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/aa41caeb809640c9839f400a70eb33c1] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp, totalSize=47.5 K 2023-03-20 00:55:35,725 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 03d073d5708c48b9ac8404b6a27e310e, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679273732332 2023-03-20 00:55:35,726 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting bccd87ae84bc4812b5092589c3ae7c21, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1679273733032 2023-03-20 00:55:35,726 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 93d6ccb109e846fdb81ab29f37053252, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1679273733696 2023-03-20 00:55:35,727 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting aa41caeb809640c9839f400a70eb33c1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1679273734064 2023-03-20 00:55:35,740 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1bfc994bd080bb1b8cbc4aef89c8cc98#C#compaction#32 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:55:35,755 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/a9db4a5e73ae48919e7b3f395d6a862f as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/a9db4a5e73ae48919e7b3f395d6a862f 2023-03-20 00:55:35,762 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1bfc994bd080bb1b8cbc4aef89c8cc98/C of 1bfc994bd080bb1b8cbc4aef89c8cc98 into a9db4a5e73ae48919e7b3f395d6a862f(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:55:35,762 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:35,762 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98., storeName=1bfc994bd080bb1b8cbc4aef89c8cc98/C, priority=12, startTime=1679273735645; duration=0sec 2023-03-20 00:55:35,762 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:55:35,763 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:35,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:35,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:35,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:35,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:35,769 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:35,769 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:35,770 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:35,770 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:35,771 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,771 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:35,772 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:35,772 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:35,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:35,773 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,773 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:35,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:35,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:35,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:35,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:35,774 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:35,774 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:35,774 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:35,774 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:35,774 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:35,776 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:35,776 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:35,776 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:35,777 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:35,777 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:35,777 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bfc994bd080bb1b8cbc4aef89c8cc98 3/3 column families, dataSize=134.18 KB heapSize=366.23 KB 2023-03-20 00:55:35,777 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=A 2023-03-20 00:55:35,777 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:35,777 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=B 2023-03-20 00:55:35,777 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:35,778 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=C 2023-03-20 00:55:35,778 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:35,804 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/6f2067f14d784cb198c1a1a7e8ce3e5b 2023-03-20 00:55:35,840 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/fe5d86fd78a74f21a643035560285986 2023-03-20 00:55:35,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:35,870 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:35,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:35,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37946 deadline: 1679273795879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:35,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:35,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37902 deadline: 1679273795879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37956 deadline: 1679273795880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:35,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37938 deadline: 1679273795886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:35,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37918 deadline: 1679273795918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:35,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37946 deadline: 1679273795987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:35,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37956 deadline: 1679273795988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:35,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37902 deadline: 1679273795988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:35,990 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:35,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37938 deadline: 1679273795988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,108 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/dfa8ca2ac4254894ae06dfb291d68a88 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/dfa8ca2ac4254894ae06dfb291d68a88 2023-03-20 00:55:36,115 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1bfc994bd080bb1b8cbc4aef89c8cc98/B of 1bfc994bd080bb1b8cbc4aef89c8cc98 into dfa8ca2ac4254894ae06dfb291d68a88(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:55:36,115 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:36,115 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98., storeName=1bfc994bd080bb1b8cbc4aef89c8cc98/B, priority=12, startTime=1679273735644; duration=0sec 2023-03-20 00:55:36,116 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:55:36,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:36,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37946 deadline: 1679273796190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,193 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:36,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37902 deadline: 1679273796191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:36,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37956 deadline: 1679273796191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:36,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37938 deadline: 1679273796191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,262 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/8e575f5f53dc492782e94551a99ef4ce 2023-03-20 00:55:36,271 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/6f2067f14d784cb198c1a1a7e8ce3e5b as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/6f2067f14d784cb198c1a1a7e8ce3e5b 2023-03-20 00:55:36,281 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/6f2067f14d784cb198c1a1a7e8ce3e5b, entries=150, sequenceid=320, filesize=11.9 K 2023-03-20 00:55:36,283 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/fe5d86fd78a74f21a643035560285986 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/fe5d86fd78a74f21a643035560285986 2023-03-20 00:55:36,290 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/fe5d86fd78a74f21a643035560285986, entries=150, sequenceid=320, filesize=11.9 K 2023-03-20 00:55:36,291 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/8e575f5f53dc492782e94551a99ef4ce as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/8e575f5f53dc492782e94551a99ef4ce 2023-03-20 00:55:36,302 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/8e575f5f53dc492782e94551a99ef4ce, entries=150, sequenceid=320, filesize=11.9 K 2023-03-20 00:55:36,303 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~366.19 KB/374976, currentSize=53.67 KB/54960 for 1bfc994bd080bb1b8cbc4aef89c8cc98 in 526ms, sequenceid=320, compaction requested=false 2023-03-20 00:55:36,303 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:36,303 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:36,303 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:36,304 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:36,304 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,305 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:36,305 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:36,306 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,306 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:36,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:36,307 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,307 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:36,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:36,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:36,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:36,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:36,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:36,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f7b255b[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:36,309 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:36,309 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,310 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,310 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:36,310 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:36,310 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:36,312 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:36,312 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:36,312 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:36,317 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,318 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:36,318 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:36,318 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:36,318 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:36,318 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:36,318 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:36,318 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:36,328 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,328 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:36,328 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:36,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:36,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:36,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:36,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:36,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:36,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:36,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:36,330 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:36,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:36,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,333 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,336 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:36,336 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:36,336 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:36,337 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:36,337 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:36,337 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:36,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:36,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:36,341 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:36,341 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:36,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:36,341 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,342 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,342 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,342 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:36,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:36,442 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:36,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:36,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:36,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:36,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:36,446 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:36,446 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:36,446 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:36,446 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:36,447 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,447 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:36,448 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,448 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:36,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:36,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:36,448 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,448 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:36,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:36,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:36,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:36,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:36,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:36,451 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:36,451 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:36,451 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:36,454 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:36,454 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:36,454 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:36,455 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:36,455 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bfc994bd080bb1b8cbc4aef89c8cc98 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-20 00:55:36,455 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=A 2023-03-20 00:55:36,455 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:36,455 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=B 2023-03-20 00:55:36,455 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:36,455 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=C 2023-03-20 00:55:36,456 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:36,472 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/c7e230ae4b09455784c5f2b1197f4d8f 2023-03-20 00:55:36,490 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/dc645a9789a845ebbfacf846bd3f5e8c 2023-03-20 00:55:36,497 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:36,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:36,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679273736512, max timestamp=1679273736519Num uniques 859; 2023-03-20 00:55:36,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 00:55:36,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679273736516, max timestamp=1679273736522Num uniques 859; 2023-03-20 00:55:36,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 00:55:36,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679273736516, max timestamp=1679273736522Num uniques 859; 2023-03-20 00:55:36,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 00:55:36,543 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/5a71ef4d830a48d9b73029c2f6f5501a 2023-03-20 00:55:36,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:36,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37902 deadline: 1679273796545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,552 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:36,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37938 deadline: 1679273796547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,552 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:36,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37956 deadline: 1679273796547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:36,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37946 deadline: 1679273796550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,554 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/c7e230ae4b09455784c5f2b1197f4d8f as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/c7e230ae4b09455784c5f2b1197f4d8f 2023-03-20 00:55:36,563 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/c7e230ae4b09455784c5f2b1197f4d8f, entries=100, sequenceid=332, filesize=9.5 K 2023-03-20 00:55:36,565 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/dc645a9789a845ebbfacf846bd3f5e8c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/dc645a9789a845ebbfacf846bd3f5e8c 2023-03-20 00:55:36,573 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/dc645a9789a845ebbfacf846bd3f5e8c, entries=100, sequenceid=332, filesize=9.5 K 2023-03-20 00:55:36,574 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/5a71ef4d830a48d9b73029c2f6f5501a as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/5a71ef4d830a48d9b73029c2f6f5501a 2023-03-20 00:55:36,589 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/5a71ef4d830a48d9b73029c2f6f5501a, entries=100, sequenceid=332, filesize=9.5 K 2023-03-20 00:55:36,591 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=129.71 KB/132820 for 1bfc994bd080bb1b8cbc4aef89c8cc98 in 135ms, sequenceid=332, compaction requested=true 2023-03-20 00:55:36,591 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:36,591 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:36,591 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:36,592 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:36,593 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,593 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:36,593 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:36,594 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:36,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:36,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:36,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:36,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:36,596 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,597 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,597 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:36,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:36,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:36,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@730144b3[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:36,598 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:36,598 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,601 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,601 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:36,601 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,601 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:36,601 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:36,601 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:36,602 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:36,602 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,602 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:36,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:36,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:36,602 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:36,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:36,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:36,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:36,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:36,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:36,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:36,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:36,605 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:36,605 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:36,605 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:36,605 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:36,605 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:36,607 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:36,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:36,608 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,608 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:36,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:36,608 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:36,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:36,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:36,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,610 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:36,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:36,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:36,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:36,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,614 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:36,614 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:36,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:36,614 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,614 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:36,614 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:36,614 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,614 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,614 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,614 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:36,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:36,614 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:36,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:36,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:36,615 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:36,615 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:36,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:36,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:36,654 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1bfc994bd080bb1b8cbc4aef89c8cc98 3/3 column families, dataSize=136.42 KB heapSize=390.06 KB 2023-03-20 00:55:36,654 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=A 2023-03-20 00:55:36,654 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:36,654 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=B 2023-03-20 00:55:36,654 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:36,655 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=C 2023-03-20 00:55:36,655 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:36,668 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:36,668 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=369 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/aa138909118345298b23c635e9f1522d 2023-03-20 00:55:36,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37956 deadline: 1679273796663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:36,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37946 deadline: 1679273796663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:36,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37938 deadline: 1679273796664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:36,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37902 deadline: 1679273796668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,688 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=369 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/6b4da5d1bde249e383c84a314e843b72 2023-03-20 00:55:36,703 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=369 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/7b2193d932cb4f0c8e68877151620d7f 2023-03-20 00:55:36,711 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/aa138909118345298b23c635e9f1522d as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/aa138909118345298b23c635e9f1522d 2023-03-20 00:55:36,715 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:36,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:36,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:36,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:36,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:36,720 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:36,720 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:36,720 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:36,720 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:36,721 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,721 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:36,721 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/aa138909118345298b23c635e9f1522d, entries=200, sequenceid=369, filesize=14.3 K 2023-03-20 00:55:36,722 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,722 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:36,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:36,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:36,722 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,722 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:36,723 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/6b4da5d1bde249e383c84a314e843b72 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/6b4da5d1bde249e383c84a314e843b72 2023-03-20 00:55:36,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:36,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:36,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:36,724 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:36,725 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:36,725 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:36,725 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:36,725 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:36,725 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:36,725 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:36,725 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:36,725 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:36,725 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:36,725 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:36,726 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:36,726 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:36,726 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:36,726 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,726 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:36,726 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:36,727 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,727 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:36,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:36,727 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,727 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:36,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:36,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:36,731 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/6b4da5d1bde249e383c84a314e843b72, entries=150, sequenceid=369, filesize=11.9 K 2023-03-20 00:55:36,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:36,734 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/7b2193d932cb4f0c8e68877151620d7f as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/7b2193d932cb4f0c8e68877151620d7f 2023-03-20 00:55:36,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:36,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:36,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@673d6b0a[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:36,736 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:36,736 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,744 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/7b2193d932cb4f0c8e68877151620d7f, entries=150, sequenceid=369, filesize=11.9 K 2023-03-20 00:55:36,744 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,745 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,745 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:36,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,745 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:36,745 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:36,745 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~136.42 KB/139690, heapSize ~390.02 KB/399376, currentSize=46.96 KB/48090 for 1bfc994bd080bb1b8cbc4aef89c8cc98 in 91ms, sequenceid=369, compaction requested=true 2023-03-20 00:55:36,745 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:36,747 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:36,747 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:55:36,747 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 00:55:36,747 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 00:55:36,747 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:55:36,747 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:55:36,749 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49082 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 00:55:36,749 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 1bfc994bd080bb1b8cbc4aef89c8cc98/A is initiating minor compaction (all files) 2023-03-20 00:55:36,749 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bfc994bd080bb1b8cbc4aef89c8cc98/A in TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:36,749 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/a1a08a26289243bbaa20ca0fecaf3ed8, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/6f2067f14d784cb198c1a1a7e8ce3e5b, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/c7e230ae4b09455784c5f2b1197f4d8f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/aa138909118345298b23c635e9f1522d] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp, totalSize=47.9 K 2023-03-20 00:55:36,749 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46642 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 00:55:36,750 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 1bfc994bd080bb1b8cbc4aef89c8cc98/B is initiating minor compaction (all files) 2023-03-20 00:55:36,750 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bfc994bd080bb1b8cbc4aef89c8cc98/B in TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:36,750 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/dfa8ca2ac4254894ae06dfb291d68a88, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/fe5d86fd78a74f21a643035560285986, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/dc645a9789a845ebbfacf846bd3f5e8c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/6b4da5d1bde249e383c84a314e843b72] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp, totalSize=45.5 K 2023-03-20 00:55:36,750 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting a1a08a26289243bbaa20ca0fecaf3ed8, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1679273734064 2023-03-20 00:55:36,750 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting dfa8ca2ac4254894ae06dfb291d68a88, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1679273734064 2023-03-20 00:55:36,753 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:36,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:36,753 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:36,753 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:36,753 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:36,755 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting fe5d86fd78a74f21a643035560285986, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1679273734785 2023-03-20 00:55:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:36,755 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 6f2067f14d784cb198c1a1a7e8ce3e5b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1679273734785 2023-03-20 00:55:36,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:36,756 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting dc645a9789a845ebbfacf846bd3f5e8c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1679273735877 2023-03-20 00:55:36,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,757 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 6b4da5d1bde249e383c84a314e843b72, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1679273736544 2023-03-20 00:55:36,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:36,758 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting c7e230ae4b09455784c5f2b1197f4d8f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1679273735877 2023-03-20 00:55:36,760 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting aa138909118345298b23c635e9f1522d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1679273736540 2023-03-20 00:55:36,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:36,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:36,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:36,763 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:36,763 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:36,763 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:36,763 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:36,763 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:36,765 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,765 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:36,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,767 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:36,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:36,767 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:36,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:36,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:36,768 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:36,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:36,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:36,771 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:36,771 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,771 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:36,771 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:36,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:36,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:36,771 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:36,771 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,771 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,771 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,771 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,771 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:36,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:36,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:36,772 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:36,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:36,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:36,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:36,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:36,774 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1bfc994bd080bb1b8cbc4aef89c8cc98 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-20 00:55:36,774 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=A 2023-03-20 00:55:36,775 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:36,775 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=B 2023-03-20 00:55:36,775 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:36,775 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=C 2023-03-20 00:55:36,776 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:36,785 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1bfc994bd080bb1b8cbc4aef89c8cc98#A#compaction#42 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:55:36,786 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 1bfc994bd080bb1b8cbc4aef89c8cc98#B#compaction#43 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:55:36,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679273736793, max timestamp=1679273736807Num uniques 859; 2023-03-20 00:55:36,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679273736793, max timestamp=1679273736807Num uniques 859; 2023-03-20 00:55:36,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 00:55:36,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679273736793, max timestamp=1679273736807Num uniques 859; 2023-03-20 00:55:36,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 00:55:36,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 00:55:36,841 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:36,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37902 deadline: 1679273796835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:36,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37956 deadline: 1679273796836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:36,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37946 deadline: 1679273796837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,846 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:36,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37938 deadline: 1679273796837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,852 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/4f5ee2a2b7bf449fb25e69a5e377deed as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/4f5ee2a2b7bf449fb25e69a5e377deed 2023-03-20 00:55:36,852 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/0d48ea45d4e447c193cb3c322ee72eb9 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/0d48ea45d4e447c193cb3c322ee72eb9 2023-03-20 00:55:36,860 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1bfc994bd080bb1b8cbc4aef89c8cc98/A of 1bfc994bd080bb1b8cbc4aef89c8cc98 into 4f5ee2a2b7bf449fb25e69a5e377deed(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:55:36,862 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1bfc994bd080bb1b8cbc4aef89c8cc98/B of 1bfc994bd080bb1b8cbc4aef89c8cc98 into 0d48ea45d4e447c193cb3c322ee72eb9(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:55:36,862 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:36,864 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:36,864 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98., storeName=1bfc994bd080bb1b8cbc4aef89c8cc98/A, priority=12, startTime=1679273736747; duration=0sec 2023-03-20 00:55:36,864 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98., storeName=1bfc994bd080bb1b8cbc4aef89c8cc98/B, priority=12, startTime=1679273736747; duration=0sec 2023-03-20 00:55:36,864 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:55:36,864 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:55:36,864 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 00:55:36,866 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46642 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 00:55:36,866 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 1bfc994bd080bb1b8cbc4aef89c8cc98/C is initiating minor compaction (all files) 2023-03-20 00:55:36,866 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bfc994bd080bb1b8cbc4aef89c8cc98/C in TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:36,866 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/a9db4a5e73ae48919e7b3f395d6a862f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/8e575f5f53dc492782e94551a99ef4ce, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/5a71ef4d830a48d9b73029c2f6f5501a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/7b2193d932cb4f0c8e68877151620d7f] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp, totalSize=45.5 K 2023-03-20 00:55:36,867 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting a9db4a5e73ae48919e7b3f395d6a862f, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1679273734064 2023-03-20 00:55:36,867 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 8e575f5f53dc492782e94551a99ef4ce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1679273734785 2023-03-20 00:55:36,868 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 5a71ef4d830a48d9b73029c2f6f5501a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1679273735877 2023-03-20 00:55:36,868 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 7b2193d932cb4f0c8e68877151620d7f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1679273736544 2023-03-20 00:55:36,872 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:36,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:36,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:36,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:36,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:36,877 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:36,877 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:36,877 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:36,877 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:36,877 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,877 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:36,881 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:36,881 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:36,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:36,881 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,881 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:36,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:36,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:36,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:36,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:36,884 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1bfc994bd080bb1b8cbc4aef89c8cc98#C#compaction#45 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:55:36,887 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:36,887 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:36,887 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:36,887 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:36,887 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:36,890 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:36,890 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:36,890 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:36,890 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:36,890 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:36,890 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:36,890 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:36,890 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:36,890 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,890 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:36,890 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:36,894 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,894 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:36,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:36,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:36,895 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,895 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:36,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:36,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:36,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:36,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:36,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16760ac0[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:36,897 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:36,898 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,901 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,901 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:36,901 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:36,901 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:36,901 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,901 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:36,902 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:36,902 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,902 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:36,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:36,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:36,902 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:36,903 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/fe2c78ad9d9d426e98f5323ba9bfe97d as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/fe2c78ad9d9d426e98f5323ba9bfe97d 2023-03-20 00:55:36,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:36,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:36,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:36,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:36,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:36,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:36,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:36,907 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:36,907 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:36,907 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:36,907 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:36,907 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:36,911 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,911 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:36,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:36,911 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:36,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,912 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1bfc994bd080bb1b8cbc4aef89c8cc98/C of 1bfc994bd080bb1b8cbc4aef89c8cc98 into fe2c78ad9d9d426e98f5323ba9bfe97d(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:55:36,912 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:36,912 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98., storeName=1bfc994bd080bb1b8cbc4aef89c8cc98/C, priority=12, startTime=1679273736747; duration=0sec 2023-03-20 00:55:36,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:36,912 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:55:36,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:36,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:36,912 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:36,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:36,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:36,913 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:36,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:36,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,915 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,915 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:36,915 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:36,915 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,915 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,915 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:36,915 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:36,915 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:36,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:36,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:36,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:36,915 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:36,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:36,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:36,915 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:36,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:36,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:36,916 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:36,916 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:36,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:36,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37918 deadline: 1679273796921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,945 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:36,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37902 deadline: 1679273796944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:36,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37956 deadline: 1679273796950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,953 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:36,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273796952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:36,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:36,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37938 deadline: 1679273796952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,016 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:37,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:37,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:37,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:37,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:37,020 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:37,020 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:37,020 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:37,020 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:37,020 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:37,020 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:37,021 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:37,021 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:37,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:37,021 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,022 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:37,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:37,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:37,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:37,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:37,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:37,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:37,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:37,023 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:37,023 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:37,023 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:37,023 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:37,023 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:37,023 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:37,023 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:37,023 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:37,023 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:37,023 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:37,023 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:37,023 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,023 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:37,023 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:37,024 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,024 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:37,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:37,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:37,025 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:37,025 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:37,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:37,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:37,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:37,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:37,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:37,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40f1b4a8[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:37,026 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:37,026 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:37,027 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:37,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:37,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:37,027 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:37,027 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:37,027 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:37,027 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,027 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:37,028 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,028 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:37,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:37,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:37,028 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:37,028 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:37,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:37,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:37,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:37,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:37,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:37,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:37,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:37,030 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:37,030 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:37,030 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:37,031 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:37,031 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:37,032 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:37,032 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:37,032 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:37,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:37,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:37,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:37,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:37,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:37,032 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:37,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:37,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:37,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:37,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:37,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:37,033 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:37,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:37,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:37,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,036 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,036 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:37,036 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:37,036 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:37,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:37,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:37,036 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,036 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:37,036 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:37,037 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:37,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:37,037 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:37,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:37,037 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:37,037 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:37,037 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:37,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:37,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:37,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:37,137 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:37,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:37,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:37,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:37,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37902 deadline: 1679273797146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:37,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:37,150 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:37,150 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:37,150 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:37,150 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:37,150 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:37,151 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:37,151 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:37,151 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:37,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:37,152 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,152 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:37,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:37,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:37,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:37,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:37,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:37,152 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:37,152 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:37,153 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:37,153 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:37,153 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:37,153 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:37,153 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:37,153 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:37,153 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:37,153 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:37,153 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:37,153 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:37,153 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:37,153 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,153 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:37,153 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:37,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:37,154 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,154 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:37,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:37,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:37,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37956 deadline: 1679273797153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,155 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:37,155 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:37,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:37,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:37,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:37,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:37,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:37,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39ac44d9[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:37,156 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:37,156 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:37,157 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:37,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:37,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:37,157 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:37,157 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:37,157 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:37,157 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,157 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:37,158 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:37,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37938 deadline: 1679273797156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,158 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:37,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:37,158 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:37,158 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:37,158 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:37,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:37,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:37,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:37,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:37,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:37,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:37,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:37,160 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:37,160 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:37,160 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:37,161 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:37,161 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:37,162 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:37,162 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:37,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:37,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:37,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:37,162 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:37,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:37,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:37,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:37,162 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:37,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:37,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:37,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:37,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:37,164 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:37,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273797163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,164 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:37,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:37,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:37,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,167 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,167 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:37,167 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:37,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:37,167 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:37,167 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:37,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:37,167 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:37,167 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,167 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:37,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:37,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:37,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:37,167 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:37,167 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:37,167 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:37,167 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:37,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:37,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:37,212 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/94942bd149a94d3f965897873a7eeaa0 2023-03-20 00:55:37,234 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/2aab22b4007b46e5921ba0708fcfd3a8 2023-03-20 00:55:37,249 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/58b3869641fe44fe9ed04c58a3ac0e99 2023-03-20 00:55:37,256 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/94942bd149a94d3f965897873a7eeaa0 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/94942bd149a94d3f965897873a7eeaa0 2023-03-20 00:55:37,262 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/94942bd149a94d3f965897873a7eeaa0, entries=200, sequenceid=380, filesize=14.3 K 2023-03-20 00:55:37,263 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/2aab22b4007b46e5921ba0708fcfd3a8 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/2aab22b4007b46e5921ba0708fcfd3a8 2023-03-20 00:55:37,268 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:37,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:37,271 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/2aab22b4007b46e5921ba0708fcfd3a8, entries=150, sequenceid=380, filesize=11.9 K 2023-03-20 00:55:37,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:37,273 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/58b3869641fe44fe9ed04c58a3ac0e99 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/58b3869641fe44fe9ed04c58a3ac0e99 2023-03-20 00:55:37,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:37,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:37,282 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:37,282 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:37,282 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:37,282 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:37,282 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:37,282 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:37,283 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:37,283 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:37,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:37,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:37,285 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,285 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:37,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:37,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:37,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:37,286 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:37,286 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:37,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:37,287 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:37,287 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:37,287 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:37,287 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:37,287 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:37,287 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:37,287 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:37,287 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:37,287 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:37,287 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:37,287 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:37,287 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,287 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:37,287 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:37,289 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:37,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:37,289 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:37,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:37,289 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:37,289 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:37,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:37,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:37,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:37,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:37,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35451ddd[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:37,291 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/58b3869641fe44fe9ed04c58a3ac0e99, entries=150, sequenceid=380, filesize=11.9 K 2023-03-20 00:55:37,291 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:37,291 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:37,292 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:37,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:37,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:37,293 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 1bfc994bd080bb1b8cbc4aef89c8cc98 in 519ms, sequenceid=380, compaction requested=false 2023-03-20 00:55:37,293 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:37,293 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:37,293 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:37,293 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,293 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:37,293 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:37,299 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:37,299 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:37,299 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:37,299 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:37,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:37,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:37,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:37,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:37,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:37,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:37,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:37,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:37,302 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:37,302 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:37,302 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:37,302 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:37,302 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:37,303 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:37,303 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:37,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:37,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:37,303 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:37,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:37,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:37,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:37,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:37,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:37,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:37,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:37,304 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:37,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:37,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:37,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,305 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:37,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:37,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,307 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,308 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:37,308 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:37,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:37,308 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:37,308 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:37,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:37,308 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:37,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:37,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:37,308 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,308 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:37,308 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:37,308 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:37,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:37,308 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:37,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:37,308 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:37,409 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:37,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:37,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:37,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:37,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:37,414 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:37,414 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:37,414 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:37,414 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:37,415 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:37,415 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:37,416 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:37,416 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:37,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:37,416 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:37,416 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:37,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:37,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:37,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:37,417 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:37,417 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:37,417 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:37,417 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:37,417 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:37,417 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:37,417 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:37,418 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:37,418 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:37,418 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:37,418 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bfc994bd080bb1b8cbc4aef89c8cc98 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-20 00:55:37,418 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=A 2023-03-20 00:55:37,418 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:37,418 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=B 2023-03-20 00:55:37,418 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:37,418 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=C 2023-03-20 00:55:37,419 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:37,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:37,455 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:37,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:37,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37956 deadline: 1679273797465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:37,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37902 deadline: 1679273797467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:37,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273797467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:37,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37938 deadline: 1679273797468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:37,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37956 deadline: 1679273797570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:37,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37902 deadline: 1679273797572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:37,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37938 deadline: 1679273797573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:37,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37956 deadline: 1679273797774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,779 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:37,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37902 deadline: 1679273797778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,779 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:37,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37938 deadline: 1679273797778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:37,850 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/c7e5d32c4f5c441f8f65a2a5d6bfabe7 2023-03-20 00:55:37,974 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:37,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273797973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:38,077 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:38,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:37956 deadline: 1679273798076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:38,083 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:38,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37902 deadline: 1679273798082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:38,083 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:38,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37938 deadline: 1679273798082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:38,265 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/43460b6977bb40b5a7e065910721a998 2023-03-20 00:55:38,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:38,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37956 deadline: 1679273798580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:38,588 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:38,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37938 deadline: 1679273798588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:38,589 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:38,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37902 deadline: 1679273798588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:38,700 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/a454bd679290463a85e2f320753deca1 2023-03-20 00:55:38,706 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/c7e5d32c4f5c441f8f65a2a5d6bfabe7 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/c7e5d32c4f5c441f8f65a2a5d6bfabe7 2023-03-20 00:55:38,713 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/c7e5d32c4f5c441f8f65a2a5d6bfabe7, entries=150, sequenceid=419, filesize=11.9 K 2023-03-20 00:55:38,714 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/43460b6977bb40b5a7e065910721a998 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/43460b6977bb40b5a7e065910721a998 2023-03-20 00:55:38,721 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/43460b6977bb40b5a7e065910721a998, entries=150, sequenceid=419, filesize=11.9 K 2023-03-20 00:55:38,723 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/a454bd679290463a85e2f320753deca1 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/a454bd679290463a85e2f320753deca1 2023-03-20 00:55:38,728 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/a454bd679290463a85e2f320753deca1, entries=150, sequenceid=419, filesize=11.9 K 2023-03-20 00:55:38,729 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 1bfc994bd080bb1b8cbc4aef89c8cc98 in 1311ms, sequenceid=419, compaction requested=true 2023-03-20 00:55:38,730 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:38,730 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:38,730 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:38,730 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:38,730 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:38,730 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:38,730 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:38,731 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:38,731 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:38,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:38,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:38,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:38,731 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:38,731 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:38,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:38,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:38,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:38,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:38,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:38,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:38,732 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:38,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@134321bf[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:38,733 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:38,733 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:38,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:38,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:38,733 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:38,733 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:38,733 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:38,733 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:38,733 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:38,734 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:38,734 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:38,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:38,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:38,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:38,734 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:38,734 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:38,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:38,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:38,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:38,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:38,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:38,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:38,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:38,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:38,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:38,737 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:38,737 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:38,737 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:38,737 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:38,737 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:38,738 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:38,738 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:38,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:38,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:38,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:38,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:38,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:38,738 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:38,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:38,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:38,739 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:38,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:38,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:38,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:38,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:38,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:38,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:38,740 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:38,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:38,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:38,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:38,742 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:38,742 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:38,742 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:38,742 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:38,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:38,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:38,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:38,742 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:38,742 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:38,742 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:38,742 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:38,742 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:38,742 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:38,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:38,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:38,742 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:38,743 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:38,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:38,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:38,842 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:38,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:38,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:38,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:38,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:38,866 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:38,866 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:38,866 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:38,866 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:38,866 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:38,866 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:38,868 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:38,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:38,868 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:38,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:38,869 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:38,869 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:38,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:38,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:38,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:38,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:38,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:38,870 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:38,870 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:38,870 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:38,870 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:38,870 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:38,874 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:38,874 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:38,874 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:38,874 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:38,874 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bfc994bd080bb1b8cbc4aef89c8cc98 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-20 00:55:38,875 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=A 2023-03-20 00:55:38,875 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:38,875 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=B 2023-03-20 00:55:38,875 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:38,875 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=C 2023-03-20 00:55:38,875 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:38,892 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/de0967b2dbb3414d88e7a693227785be 2023-03-20 00:55:38,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:38,942 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:38,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679273738983, max timestamp=1679273738992Num uniques 859; 2023-03-20 00:55:38,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679273738983, max timestamp=1679273738992Num uniques 859; 2023-03-20 00:55:38,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 00:55:38,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 00:55:38,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679273738983, max timestamp=1679273738992Num uniques 859; 2023-03-20 00:55:38,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 00:55:39,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:39,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273799032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:39,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273799033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:39,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273799136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:39,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273799138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,322 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/410016499759442c99ad10333e600121 2023-03-20 00:55:39,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:39,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273799346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,349 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:39,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273799346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,368 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/8ad87668756b4df79d3bdf98c265902a 2023-03-20 00:55:39,377 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/de0967b2dbb3414d88e7a693227785be as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/de0967b2dbb3414d88e7a693227785be 2023-03-20 00:55:39,389 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/de0967b2dbb3414d88e7a693227785be, entries=150, sequenceid=430, filesize=11.9 K 2023-03-20 00:55:39,391 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/410016499759442c99ad10333e600121 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/410016499759442c99ad10333e600121 2023-03-20 00:55:39,399 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/410016499759442c99ad10333e600121, entries=150, sequenceid=430, filesize=11.9 K 2023-03-20 00:55:39,401 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/8ad87668756b4df79d3bdf98c265902a as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/8ad87668756b4df79d3bdf98c265902a 2023-03-20 00:55:39,432 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/8ad87668756b4df79d3bdf98c265902a, entries=150, sequenceid=430, filesize=11.9 K 2023-03-20 00:55:39,433 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 1bfc994bd080bb1b8cbc4aef89c8cc98 in 559ms, sequenceid=430, compaction requested=true 2023-03-20 00:55:39,433 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:39,433 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:39,433 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:39,433 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:39,433 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:39,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:39,435 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:39,435 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:39,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:39,435 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:39,435 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:39,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:39,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:39,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:39,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:39,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:39,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5eda938b[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:39,437 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:39,438 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:39,439 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:39,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:39,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:39,439 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:39,439 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:39,439 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:39,439 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,439 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:39,440 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:39,440 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:39,440 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:39,440 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:39,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:39,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:39,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:39,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:39,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:39,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:39,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:39,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:39,445 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:39,445 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:39,445 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:39,445 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:39,445 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:39,461 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:39,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:39,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:39,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:39,462 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:39,462 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:39,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:39,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:39,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:39,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:39,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:39,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:39,463 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:39,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:39,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:39,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,465 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:39,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:39,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,470 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:39,470 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:39,470 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:39,470 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:39,470 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,470 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:39,470 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:39,470 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:39,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:39,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:39,470 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:39,470 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:39,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:39,470 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:39,470 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:39,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:39,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:39,570 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:39,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:39,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:39,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:39,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:39,575 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:39,575 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:39,575 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:39,575 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:39,577 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:39,577 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:39,578 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:39,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:39,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:39,578 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:39,578 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,578 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:39,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:39,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:39,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:39,583 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:39,583 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:39,583 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:39,583 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:39,583 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:39,583 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:39,583 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:39,583 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:39,583 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:39,583 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:39,583 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bfc994bd080bb1b8cbc4aef89c8cc98 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-20 00:55:39,584 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=A 2023-03-20 00:55:39,584 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:39,584 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=B 2023-03-20 00:55:39,584 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:39,584 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=C 2023-03-20 00:55:39,584 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:39,592 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:39,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:39,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:39,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37956 deadline: 1679273799605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:39,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37938 deadline: 1679273799606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,611 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:39,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37902 deadline: 1679273799609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,653 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:39,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273799650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,654 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:39,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273799653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:39,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37956 deadline: 1679273799710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:39,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37938 deadline: 1679273799711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:39,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37902 deadline: 1679273799713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:39,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37956 deadline: 1679273799919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,922 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:39,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37938 deadline: 1679273799919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:39,922 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:39,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37902 deadline: 1679273799920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:40,007 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/e1cc1a9deca74ed182ff8db5f49136be 2023-03-20 00:55:40,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:40,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273800156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:40,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:40,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273800157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:40,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:40,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37938 deadline: 1679273800224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:40,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:40,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37956 deadline: 1679273800224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:40,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:40,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37902 deadline: 1679273800228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:40,484 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/105d742d161643a9b5e0ec40345df98f 2023-03-20 00:55:40,730 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:40,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37938 deadline: 1679273800730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:40,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:40,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37956 deadline: 1679273800732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:40,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:40,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37902 deadline: 1679273800736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:40,897 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/2689a84dcc1c41a8affa864d3776e66d 2023-03-20 00:55:40,903 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/e1cc1a9deca74ed182ff8db5f49136be as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/e1cc1a9deca74ed182ff8db5f49136be 2023-03-20 00:55:40,908 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/e1cc1a9deca74ed182ff8db5f49136be, entries=150, sequenceid=466, filesize=11.9 K 2023-03-20 00:55:40,909 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/105d742d161643a9b5e0ec40345df98f as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/105d742d161643a9b5e0ec40345df98f 2023-03-20 00:55:40,917 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/105d742d161643a9b5e0ec40345df98f, entries=150, sequenceid=466, filesize=11.9 K 2023-03-20 00:55:40,920 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/2689a84dcc1c41a8affa864d3776e66d as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/2689a84dcc1c41a8affa864d3776e66d 2023-03-20 00:55:40,926 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/2689a84dcc1c41a8affa864d3776e66d, entries=150, sequenceid=466, filesize=11.9 K 2023-03-20 00:55:40,926 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 1bfc994bd080bb1b8cbc4aef89c8cc98 in 1343ms, sequenceid=466, compaction requested=true 2023-03-20 00:55:40,927 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:40,927 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:40,927 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:40,927 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:40,927 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:40,927 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:40,927 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:40,928 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:40,928 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:40,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:40,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:40,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:40,928 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:40,928 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:40,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:40,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:40,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:40,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:40,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:40,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:40,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50b9e05[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:40,929 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:40,929 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:40,930 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:40,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:40,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:40,930 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:40,930 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:40,930 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:40,930 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:40,930 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:40,931 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:40,931 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:40,931 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:40,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:40,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:40,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:40,931 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:40,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:40,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:40,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:40,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:40,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:40,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:40,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:40,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:40,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:40,933 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:40,934 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:40,934 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:40,934 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:40,934 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:40,935 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:40,935 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:40,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:40,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:40,935 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:40,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:40,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:40,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:40,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:40,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:40,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:40,936 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:40,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:40,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:40,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:40,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:40,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:40,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:40,937 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:40,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:40,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:40,939 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:40,940 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:40,939 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:40,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:40,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:40,940 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:40,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:40,940 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:40,940 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:40,940 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:40,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:40,940 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:40,940 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:40,940 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:40,940 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:40,940 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:40,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:40,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:41,040 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:41,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:41,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:41,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:41,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:41,044 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:41,044 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:41,044 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:41,044 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:41,044 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:41,044 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:41,045 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:41,045 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:41,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:41,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:41,045 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:41,045 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:41,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:41,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:41,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:41,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:41,046 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:41,046 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:41,046 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:41,046 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:41,046 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:41,046 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:41,047 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:41,047 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:41,047 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:41,047 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:41,047 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bfc994bd080bb1b8cbc4aef89c8cc98 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-20 00:55:41,047 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=A 2023-03-20 00:55:41,047 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:41,047 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=B 2023-03-20 00:55:41,047 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:41,047 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=C 2023-03-20 00:55:41,047 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:41,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:41,170 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:41,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679273741207, max timestamp=1679273741221Num uniques 859; 2023-03-20 00:55:41,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679273741207, max timestamp=1679273741221Num uniques 859; 2023-03-20 00:55:41,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 00:55:41,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 00:55:41,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679273741207, max timestamp=1679273741221Num uniques 859; 2023-03-20 00:55:41,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 00:55:41,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:41,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273801271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:41,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:41,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273801276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:41,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:41,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273801381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:41,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:41,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273801381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:41,464 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/c4151e07b2a54a38b9fd40fe2f9b3c2c 2023-03-20 00:55:41,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:41,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273801584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:41,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:41,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273801585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:41,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:41,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37938 deadline: 1679273801734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:41,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:41,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37956 deadline: 1679273801741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:41,750 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:41,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37902 deadline: 1679273801749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:41,882 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/0d8f92ddd86f4425a736d80c425ac153 2023-03-20 00:55:41,888 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:41,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273801886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:41,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:41,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273801889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:41,894 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/5be3121dcef34621ac78d50eba01106b 2023-03-20 00:55:41,899 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/c4151e07b2a54a38b9fd40fe2f9b3c2c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/c4151e07b2a54a38b9fd40fe2f9b3c2c 2023-03-20 00:55:41,903 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/c4151e07b2a54a38b9fd40fe2f9b3c2c, entries=150, sequenceid=477, filesize=11.9 K 2023-03-20 00:55:41,904 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/0d8f92ddd86f4425a736d80c425ac153 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/0d8f92ddd86f4425a736d80c425ac153 2023-03-20 00:55:41,910 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/0d8f92ddd86f4425a736d80c425ac153, entries=150, sequenceid=477, filesize=11.9 K 2023-03-20 00:55:41,911 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/5be3121dcef34621ac78d50eba01106b as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/5be3121dcef34621ac78d50eba01106b 2023-03-20 00:55:41,916 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/5be3121dcef34621ac78d50eba01106b, entries=150, sequenceid=477, filesize=11.9 K 2023-03-20 00:55:41,917 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 1bfc994bd080bb1b8cbc4aef89c8cc98 in 870ms, sequenceid=477, compaction requested=true 2023-03-20 00:55:41,917 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:41,917 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:41,917 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:41,917 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:41,917 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:41,917 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:41,917 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:41,918 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:41,918 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:41,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:41,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:41,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:41,919 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:41,919 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:41,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:41,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:41,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:41,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:41,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:41,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:41,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@183dc08c[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:41,923 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:41,923 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:41,924 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:41,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:41,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:41,924 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:41,924 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:41,924 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:41,924 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:41,924 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:41,926 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:41,926 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:41,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:41,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:41,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:41,926 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:41,926 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:41,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:41,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:41,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:41,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:41,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:41,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:41,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:41,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:41,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:41,928 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:41,928 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:41,928 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:41,928 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:41,928 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:41,929 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:41,929 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:41,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:41,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:41,929 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:41,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:41,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:41,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:41,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:41,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:41,929 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:41,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:41,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:41,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:41,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:41,931 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:41,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:41,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:41,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:41,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:41,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:41,933 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:41,933 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:41,933 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:41,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:41,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:41,933 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:41,933 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:41,933 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:41,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:41,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:41,933 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:41,933 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:41,933 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:41,933 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:41,933 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:41,933 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:41,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:41,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:41,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:42,033 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:42,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:42,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:42,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:42,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:42,037 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:42,037 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:42,037 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:42,037 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:42,037 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:42,037 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:42,038 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,038 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:42,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:42,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:42,038 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,038 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:42,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:42,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:42,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:42,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:42,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:42,045 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:42,045 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:42,045 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:42,045 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:42,045 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:42,045 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:42,045 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:42,045 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:42,045 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bfc994bd080bb1b8cbc4aef89c8cc98 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-20 00:55:42,046 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=A 2023-03-20 00:55:42,046 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:42,046 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=B 2023-03-20 00:55:42,046 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:42,046 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=C 2023-03-20 00:55:42,046 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:42,059 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/7fefe4cd442f4027b05d32a70edd9b0c 2023-03-20 00:55:42,070 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/15ec207d262d437bba2cf616e8e5b1cd 2023-03-20 00:55:42,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:42,398 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:42,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:42,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273802406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,413 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:42,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273802408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,486 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/9516b841e1064e89bc4bf56db525d658 2023-03-20 00:55:42,499 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/7fefe4cd442f4027b05d32a70edd9b0c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/7fefe4cd442f4027b05d32a70edd9b0c 2023-03-20 00:55:42,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:42,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273802510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:42,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273802515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,536 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/7fefe4cd442f4027b05d32a70edd9b0c, entries=150, sequenceid=512, filesize=11.9 K 2023-03-20 00:55:42,537 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/15ec207d262d437bba2cf616e8e5b1cd as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/15ec207d262d437bba2cf616e8e5b1cd 2023-03-20 00:55:42,550 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/15ec207d262d437bba2cf616e8e5b1cd, entries=150, sequenceid=512, filesize=11.9 K 2023-03-20 00:55:42,551 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/9516b841e1064e89bc4bf56db525d658 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/9516b841e1064e89bc4bf56db525d658 2023-03-20 00:55:42,557 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/9516b841e1064e89bc4bf56db525d658, entries=150, sequenceid=512, filesize=11.9 K 2023-03-20 00:55:42,559 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 1bfc994bd080bb1b8cbc4aef89c8cc98 in 513ms, sequenceid=512, compaction requested=true 2023-03-20 00:55:42,559 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:42,559 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:42,559 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:42,559 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:42,559 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,559 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:42,559 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:42,561 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,562 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:42,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:42,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:42,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:42,563 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:42,563 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:42,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:42,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:42,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:42,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:42,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12f944ed[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:42,564 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:42,564 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:42,565 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:42,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:42,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:42,565 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:42,565 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:42,565 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:42,565 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,565 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:42,566 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:42,566 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:42,566 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:42,566 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:42,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:42,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:42,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:42,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:42,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:42,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:42,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:42,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:42,569 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:42,569 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:42,569 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:42,569 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:42,569 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:42,576 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:42,576 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:42,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:42,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:42,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:42,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:42,576 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:42,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:42,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:42,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:42,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:42,577 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:42,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:42,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:42,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:42,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,578 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:42,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:42,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,584 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:42,585 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:42,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:42,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:42,585 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,585 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:42,585 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:42,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:42,585 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:42,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:42,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:42,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:42,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:42,585 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,585 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:42,585 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:42,585 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:42,585 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:42,585 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:42,686 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:42,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:42,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:42,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:42,717 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1bfc994bd080bb1b8cbc4aef89c8cc98 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-20 00:55:42,717 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=A 2023-03-20 00:55:42,717 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:42,717 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=B 2023-03-20 00:55:42,717 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:42,717 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=C 2023-03-20 00:55:42,718 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:42,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:42,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:42,720 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:42,720 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:42,720 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:42,720 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:42,721 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:42,721 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:42,722 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,722 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:42,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:42,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:42,723 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,723 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:42,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:42,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:42,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:42,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:42,724 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:42,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:42,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:42,736 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:42,736 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:42,736 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:42,742 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:42,742 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:42,742 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:42,742 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:42,742 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:42,742 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:42,742 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:42,743 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:42,743 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,754 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:42,754 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:42,755 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,755 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:42,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:42,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:42,756 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:42,756 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:42,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:42,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:42,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:42,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:42,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:42,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1835cbf2[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:42,758 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:42,758 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:42,759 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:42,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:42,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:42,759 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:42,759 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:42,759 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:42,759 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,759 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:42,765 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:42,765 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:42,765 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:42,765 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:42,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:42,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:42,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:42,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:42,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:42,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:42,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:42,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:42,769 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:42,769 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:42,769 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:42,769 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:42,769 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:42,778 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:42,778 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:42,778 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:42,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:42,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:42,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:42,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:42,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:42,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:42,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:42,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:42,779 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:42,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:42,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:42,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:42,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,780 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:42,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:42,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,786 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,786 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:42,786 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:42,786 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,786 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:42,786 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:42,786 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:42,786 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:42,786 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:42,787 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:42,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:42,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:42,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:42,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:42,787 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:42,787 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:42,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:42,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:42,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:42,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273802786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:42,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273802790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,798 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=524 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/861359865da649c1a36ef859679b177e 2023-03-20 00:55:42,841 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=524 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/ee0877aad135400ea931cc7b17d114cf 2023-03-20 00:55:42,887 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:42,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:42,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:42,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:42,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273802892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:42,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273802897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:42,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:42,905 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:42,905 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:42,905 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:42,905 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:42,906 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:42,906 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:42,907 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:42,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:42,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:42,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:42,907 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:42,908 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,908 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:42,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:42,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:42,908 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:42,918 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:42,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:42,918 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:42,918 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:42,918 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:42,920 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:42,920 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:42,920 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:42,920 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:42,920 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:42,921 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:42,921 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:42,921 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:42,921 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,921 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:42,921 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:42,922 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:42,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:42,922 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:42,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:42,923 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:42,923 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:42,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:42,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:42,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:42,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:42,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41979547[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:42,925 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:42,925 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:42,927 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,927 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:42,928 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:42,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:42,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:42,928 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:42,928 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:42,928 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:42,936 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,936 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:42,936 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:42,936 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:42,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:42,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:42,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:42,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:42,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:42,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:42,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:42,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:42,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:42,939 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:42,939 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:42,939 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:42,940 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:42,940 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:42,944 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:42,944 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:42,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:42,945 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:42,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:42,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:42,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:42,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:42,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:42,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:42,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:42,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:42,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:42,955 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:42,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:42,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,956 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:42,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:42,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,967 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:42,967 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:42,967 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:42,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:42,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:42,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:42,967 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:42,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:42,968 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:42,968 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:42,968 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:42,968 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:42,968 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:42,968 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:42,968 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:42,968 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:42,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:42,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:43,068 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:43,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:43,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:43,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:43,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:43,073 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:43,073 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:43,073 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:43,073 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:43,073 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:43,073 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:43,074 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:43,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:43,074 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:43,075 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,075 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:43,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:43,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:43,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:43,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:43,075 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:43,075 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:43,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:43,076 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:43,076 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:43,076 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:43,077 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:43,077 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:43,077 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:43,077 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:43,077 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:43,077 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:43,077 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:43,077 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:43,077 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,077 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:43,077 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:43,078 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,078 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:43,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:43,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:43,078 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:43,078 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:43,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:43,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:43,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:43,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:43,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:43,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37d81459[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:43,080 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:43,080 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:43,080 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:43,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:43,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:43,080 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:43,080 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:43,080 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:43,080 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,081 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:43,082 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:43,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:43,082 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:43,082 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:43,082 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:43,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:43,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:43,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:43,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:43,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:43,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:43,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:43,086 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:43,086 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:43,086 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:43,086 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:43,086 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:43,087 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:43,087 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:43,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:43,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:43,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:43,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:43,087 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:43,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:43,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:43,087 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:43,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:43,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:43,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:43,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:43,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:43,088 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:43,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:43,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,090 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,090 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:43,090 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:43,090 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:43,090 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:43,090 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:43,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:43,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:43,090 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:43,090 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:43,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:43,090 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:43,090 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:43,090 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:43,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:43,091 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:43,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:43,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:43,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:43,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273803095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:43,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273803103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,191 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:43,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:43,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:43,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:43,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:43,194 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:43,194 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:43,194 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:43,194 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:43,195 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:43,195 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:43,195 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:43,195 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:43,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:43,196 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,196 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:43,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:43,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:43,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:43,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:43,196 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:43,197 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:43,197 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:43,197 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:43,197 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:43,197 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:43,199 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:43,199 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:43,199 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:43,199 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:43,199 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:43,199 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:43,199 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:43,199 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:43,199 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,199 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:43,199 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:43,200 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,200 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:43,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:43,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:43,201 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:43,201 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:43,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:43,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:43,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:43,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:43,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:43,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fdef063[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:43,202 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:43,202 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:43,203 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:43,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:43,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:43,203 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:43,203 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:43,203 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:43,203 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,203 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:43,204 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,204 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:43,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:43,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:43,204 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:43,204 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:43,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:43,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:43,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:43,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:43,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:43,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:43,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:43,205 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:43,206 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:43,206 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:43,206 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:43,206 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:43,208 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:43,208 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:43,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:43,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:43,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:43,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:43,208 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:43,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:43,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:43,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:43,209 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:43,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:43,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:43,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:43,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:43,210 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:43,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:43,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,211 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,211 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:43,211 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:43,212 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,212 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:43,212 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:43,212 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:43,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:43,212 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:43,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:43,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:43,212 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:43,212 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:43,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:43,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:43,212 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:43,213 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:43,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:43,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:43,278 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=524 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/129ef095c05f4166be5cf23a02c40b1e 2023-03-20 00:55:43,287 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/861359865da649c1a36ef859679b177e as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/861359865da649c1a36ef859679b177e 2023-03-20 00:55:43,293 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/861359865da649c1a36ef859679b177e, entries=200, sequenceid=524, filesize=14.3 K 2023-03-20 00:55:43,294 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/ee0877aad135400ea931cc7b17d114cf as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/ee0877aad135400ea931cc7b17d114cf 2023-03-20 00:55:43,300 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/ee0877aad135400ea931cc7b17d114cf, entries=150, sequenceid=524, filesize=11.9 K 2023-03-20 00:55:43,301 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/129ef095c05f4166be5cf23a02c40b1e as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/129ef095c05f4166be5cf23a02c40b1e 2023-03-20 00:55:43,305 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/129ef095c05f4166be5cf23a02c40b1e, entries=150, sequenceid=524, filesize=11.9 K 2023-03-20 00:55:43,306 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 1bfc994bd080bb1b8cbc4aef89c8cc98 in 589ms, sequenceid=524, compaction requested=true 2023-03-20 00:55:43,306 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:43,306 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:55:43,306 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-03-20 00:55:43,307 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-03-20 00:55:43,307 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:55:43,307 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:55:43,310 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 102846 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-03-20 00:55:43,310 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 1bfc994bd080bb1b8cbc4aef89c8cc98/A is initiating minor compaction (all files) 2023-03-20 00:55:43,310 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 97966 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-03-20 00:55:43,310 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bfc994bd080bb1b8cbc4aef89c8cc98/A in TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:43,311 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 1bfc994bd080bb1b8cbc4aef89c8cc98/B is initiating minor compaction (all files) 2023-03-20 00:55:43,311 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bfc994bd080bb1b8cbc4aef89c8cc98/B in TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:43,311 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/4f5ee2a2b7bf449fb25e69a5e377deed, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/94942bd149a94d3f965897873a7eeaa0, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/c7e5d32c4f5c441f8f65a2a5d6bfabe7, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/de0967b2dbb3414d88e7a693227785be, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/e1cc1a9deca74ed182ff8db5f49136be, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/c4151e07b2a54a38b9fd40fe2f9b3c2c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/7fefe4cd442f4027b05d32a70edd9b0c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/861359865da649c1a36ef859679b177e] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp, totalSize=100.4 K 2023-03-20 00:55:43,311 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/0d48ea45d4e447c193cb3c322ee72eb9, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/2aab22b4007b46e5921ba0708fcfd3a8, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/43460b6977bb40b5a7e065910721a998, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/410016499759442c99ad10333e600121, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/105d742d161643a9b5e0ec40345df98f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/0d8f92ddd86f4425a736d80c425ac153, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/15ec207d262d437bba2cf616e8e5b1cd, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/ee0877aad135400ea931cc7b17d114cf] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp, totalSize=95.7 K 2023-03-20 00:55:43,311 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 4f5ee2a2b7bf449fb25e69a5e377deed, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1679273736544 2023-03-20 00:55:43,311 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 0d48ea45d4e447c193cb3c322ee72eb9, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1679273736544 2023-03-20 00:55:43,312 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 94942bd149a94d3f965897873a7eeaa0, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1679273736656 2023-03-20 00:55:43,312 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 2aab22b4007b46e5921ba0708fcfd3a8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1679273736656 2023-03-20 00:55:43,312 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting c7e5d32c4f5c441f8f65a2a5d6bfabe7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1679273736825 2023-03-20 00:55:43,312 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 43460b6977bb40b5a7e065910721a998, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1679273736825 2023-03-20 00:55:43,313 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:43,313 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting de0967b2dbb3414d88e7a693227785be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1679273737453 2023-03-20 00:55:43,313 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 410016499759442c99ad10333e600121, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1679273737453 2023-03-20 00:55:43,313 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting e1cc1a9deca74ed182ff8db5f49136be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1679273739017 2023-03-20 00:55:43,314 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 105d742d161643a9b5e0ec40345df98f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1679273739017 2023-03-20 00:55:43,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:43,314 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting c4151e07b2a54a38b9fd40fe2f9b3c2c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1679273739602 2023-03-20 00:55:43,314 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 0d8f92ddd86f4425a736d80c425ac153, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1679273739602 2023-03-20 00:55:43,315 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 7fefe4cd442f4027b05d32a70edd9b0c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1679273741261 2023-03-20 00:55:43,315 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 15ec207d262d437bba2cf616e8e5b1cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1679273741261 2023-03-20 00:55:43,315 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 861359865da649c1a36ef859679b177e, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1679273742404 2023-03-20 00:55:43,315 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting ee0877aad135400ea931cc7b17d114cf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1679273742405 2023-03-20 00:55:43,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:43,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:43,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:43,317 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:43,317 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:43,317 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:43,317 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:43,317 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:43,317 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:43,318 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:43,318 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:43,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:43,319 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,319 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:43,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:43,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:43,320 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:43,320 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:43,320 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:43,320 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:43,320 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:43,320 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:43,321 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:43,321 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:43,321 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:43,321 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:43,321 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bfc994bd080bb1b8cbc4aef89c8cc98 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-20 00:55:43,321 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=A 2023-03-20 00:55:43,321 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:43,322 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=B 2023-03-20 00:55:43,322 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:43,322 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=C 2023-03-20 00:55:43,322 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:43,340 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 1bfc994bd080bb1b8cbc4aef89c8cc98#B#compaction#67 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:55:43,358 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=544 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/331b549410754af7bae1d19679be1e8d 2023-03-20 00:55:43,358 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1bfc994bd080bb1b8cbc4aef89c8cc98#A#compaction#68 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:55:43,370 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/106fdffcb1d845d5a7b76bf4b01a4ccf as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/106fdffcb1d845d5a7b76bf4b01a4ccf 2023-03-20 00:55:43,376 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=544 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/4e9daa43cb10457293fa1e108674bbb4 2023-03-20 00:55:43,376 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 1bfc994bd080bb1b8cbc4aef89c8cc98/B of 1bfc994bd080bb1b8cbc4aef89c8cc98 into 106fdffcb1d845d5a7b76bf4b01a4ccf(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:55:43,376 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:43,377 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98., storeName=1bfc994bd080bb1b8cbc4aef89c8cc98/B, priority=8, startTime=1679273743307; duration=0sec 2023-03-20 00:55:43,377 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:55:43,377 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-03-20 00:55:43,379 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 97966 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-03-20 00:55:43,379 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 1bfc994bd080bb1b8cbc4aef89c8cc98/C is initiating minor compaction (all files) 2023-03-20 00:55:43,379 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bfc994bd080bb1b8cbc4aef89c8cc98/C in TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:43,379 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/fe2c78ad9d9d426e98f5323ba9bfe97d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/58b3869641fe44fe9ed04c58a3ac0e99, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/a454bd679290463a85e2f320753deca1, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/8ad87668756b4df79d3bdf98c265902a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/2689a84dcc1c41a8affa864d3776e66d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/5be3121dcef34621ac78d50eba01106b, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/9516b841e1064e89bc4bf56db525d658, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/129ef095c05f4166be5cf23a02c40b1e] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp, totalSize=95.7 K 2023-03-20 00:55:43,380 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting fe2c78ad9d9d426e98f5323ba9bfe97d, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1679273736544 2023-03-20 00:55:43,380 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 58b3869641fe44fe9ed04c58a3ac0e99, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1679273736656 2023-03-20 00:55:43,380 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting a454bd679290463a85e2f320753deca1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1679273736825 2023-03-20 00:55:43,381 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 8ad87668756b4df79d3bdf98c265902a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1679273737453 2023-03-20 00:55:43,381 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 2689a84dcc1c41a8affa864d3776e66d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1679273739017 2023-03-20 00:55:43,381 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 5be3121dcef34621ac78d50eba01106b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1679273739602 2023-03-20 00:55:43,382 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 9516b841e1064e89bc4bf56db525d658, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1679273741261 2023-03-20 00:55:43,382 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 129ef095c05f4166be5cf23a02c40b1e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1679273742405 2023-03-20 00:55:43,400 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 1bfc994bd080bb1b8cbc4aef89c8cc98#C#compaction#71 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:55:43,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:43,405 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:43,417 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/6cdd3bfcbb1c473fbe6b81b03b427cad as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/6cdd3bfcbb1c473fbe6b81b03b427cad 2023-03-20 00:55:43,422 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:43,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273803419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:43,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273803423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,423 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 1bfc994bd080bb1b8cbc4aef89c8cc98/C of 1bfc994bd080bb1b8cbc4aef89c8cc98 into 6cdd3bfcbb1c473fbe6b81b03b427cad(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:55:43,424 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:43,424 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98., storeName=1bfc994bd080bb1b8cbc4aef89c8cc98/C, priority=8, startTime=1679273743307; duration=0sec 2023-03-20 00:55:43,424 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:55:43,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:43,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:43,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273803524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273803524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:43,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273803728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:43,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273803728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:43,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37938 deadline: 1679273803751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,753 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4146 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98., hostname=jenkins-hbase20.apache.org,34553,1679273725199, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 00:55:43,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:43,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:43,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37956 deadline: 1679273803755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37902 deadline: 1679273803757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,762 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98., hostname=jenkins-hbase20.apache.org,34553,1679273725199, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 00:55:43,762 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4157 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98., hostname=jenkins-hbase20.apache.org,34553,1679273725199, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 00:55:43,770 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/7273e3433837438fa3ea8d1dc05d2129 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/7273e3433837438fa3ea8d1dc05d2129 2023-03-20 00:55:43,776 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 1bfc994bd080bb1b8cbc4aef89c8cc98/A of 1bfc994bd080bb1b8cbc4aef89c8cc98 into 7273e3433837438fa3ea8d1dc05d2129(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:55:43,776 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:43,776 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98., storeName=1bfc994bd080bb1b8cbc4aef89c8cc98/A, priority=8, startTime=1679273743306; duration=0sec 2023-03-20 00:55:43,777 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:55:43,796 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=544 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/99ab2e310fe24a71a493576e706e685c 2023-03-20 00:55:43,802 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/331b549410754af7bae1d19679be1e8d as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/331b549410754af7bae1d19679be1e8d 2023-03-20 00:55:43,808 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/331b549410754af7bae1d19679be1e8d, entries=150, sequenceid=544, filesize=11.9 K 2023-03-20 00:55:43,809 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/4e9daa43cb10457293fa1e108674bbb4 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/4e9daa43cb10457293fa1e108674bbb4 2023-03-20 00:55:43,815 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/4e9daa43cb10457293fa1e108674bbb4, entries=150, sequenceid=544, filesize=11.9 K 2023-03-20 00:55:43,816 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/99ab2e310fe24a71a493576e706e685c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/99ab2e310fe24a71a493576e706e685c 2023-03-20 00:55:43,822 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/99ab2e310fe24a71a493576e706e685c, entries=150, sequenceid=544, filesize=11.9 K 2023-03-20 00:55:43,823 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 1bfc994bd080bb1b8cbc4aef89c8cc98 in 502ms, sequenceid=544, compaction requested=false 2023-03-20 00:55:43,823 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:43,823 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:43,823 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:43,823 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:43,823 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,826 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:43,826 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:43,827 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,827 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:43,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:43,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:43,827 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:43,827 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:43,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:43,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:43,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:43,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:43,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:43,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5982cc31[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:43,829 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:43,829 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:43,830 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:43,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:43,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:43,830 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:43,830 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:43,830 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:43,830 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,830 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:43,831 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,831 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:43,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,831 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:43,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:43,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:43,831 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:43,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:43,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:43,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:43,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:43,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:43,833 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:43,833 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:43,833 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:43,834 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:43,834 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:43,835 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:43,835 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:43,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:43,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:43,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:43,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:43,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:43,835 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:43,836 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:43,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:43,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:43,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:43,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:43,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:43,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:43,837 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:43,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:43,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,839 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,839 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:43,839 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:43,839 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:43,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:43,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:43,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:43,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:43,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:43,840 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:43,840 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:43,840 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:43,840 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:43,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:43,840 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:43,840 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:43,840 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:43,840 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:43,940 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:43,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:43,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:43,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:43,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:43,947 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:43,947 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:43,947 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:43,947 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:43,947 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:43,947 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:43,948 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:43,948 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:43,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:43,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:43,950 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:43,950 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:43,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:43,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:43,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:43,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:43,961 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:43,961 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:43,961 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:43,961 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:43,962 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:43,962 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:43,962 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:43,962 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:43,965 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:43,965 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bfc994bd080bb1b8cbc4aef89c8cc98 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-20 00:55:43,965 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=A 2023-03-20 00:55:43,965 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:43,965 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=B 2023-03-20 00:55:43,966 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:43,966 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=C 2023-03-20 00:55:43,966 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:44,009 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=559 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/9b344171f5494d96922b03c6b8515c0c 2023-03-20 00:55:44,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:44,032 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:44,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:44,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273804067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:44,072 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:44,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273804071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:44,176 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:44,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:44,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273804172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:44,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273804173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:44,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:44,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273804378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:44,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:44,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273804378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:44,487 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=559 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/d20fd632f17c4272b4de8e4953494440 2023-03-20 00:55:44,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:44,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273804682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:44,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:44,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273804682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:44,902 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=559 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/88948025020d4bada6adb03b0a4e120d 2023-03-20 00:55:44,908 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/9b344171f5494d96922b03c6b8515c0c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/9b344171f5494d96922b03c6b8515c0c 2023-03-20 00:55:44,915 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/9b344171f5494d96922b03c6b8515c0c, entries=150, sequenceid=559, filesize=11.9 K 2023-03-20 00:55:44,916 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/d20fd632f17c4272b4de8e4953494440 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/d20fd632f17c4272b4de8e4953494440 2023-03-20 00:55:44,922 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/d20fd632f17c4272b4de8e4953494440, entries=150, sequenceid=559, filesize=11.9 K 2023-03-20 00:55:44,923 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/88948025020d4bada6adb03b0a4e120d as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/88948025020d4bada6adb03b0a4e120d 2023-03-20 00:55:44,933 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/88948025020d4bada6adb03b0a4e120d, entries=150, sequenceid=559, filesize=11.9 K 2023-03-20 00:55:44,934 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 1bfc994bd080bb1b8cbc4aef89c8cc98 in 969ms, sequenceid=559, compaction requested=true 2023-03-20 00:55:44,934 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:44,934 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:44,934 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:44,934 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:44,934 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:44,934 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:44,934 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:44,936 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:44,936 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:44,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:44,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:44,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:44,936 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:44,936 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:44,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:44,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:44,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:44,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:44,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:44,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:44,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24c257eb[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:44,939 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:44,939 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:44,940 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:44,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:44,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:44,940 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:44,940 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:44,940 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:44,941 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:44,941 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:44,941 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:44,941 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:44,941 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:44,941 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:44,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:44,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:44,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:44,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:44,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:44,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:44,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:44,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:44,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:44,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:44,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:44,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:44,944 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:44,944 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:44,944 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:44,944 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:44,944 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:44,947 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:44,948 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:44,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:44,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:44,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:44,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:44,948 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:44,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:44,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:44,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:44,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:44,948 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:44,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:44,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:44,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:44,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:44,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:44,950 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:44,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:44,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:44,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:44,954 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:44,954 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:44,954 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:44,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:44,954 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:44,954 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:44,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:44,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:44,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:44,954 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:44,955 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:44,955 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:44,955 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:44,955 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:44,955 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:44,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:44,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:44,955 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:45,055 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:45,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:45,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:45,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:45,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:45,058 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:45,058 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:45,058 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:45,059 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:45,059 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:45,060 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:45,060 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:45,060 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:45,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:45,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:45,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:45,061 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:45,061 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:45,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:45,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:45,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:45,062 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:45,062 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:45,062 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:45,062 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:45,062 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:45,062 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:45,062 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:45,062 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:45,063 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:45,063 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:45,063 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bfc994bd080bb1b8cbc4aef89c8cc98 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-20 00:55:45,063 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=A 2023-03-20 00:55:45,064 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:45,064 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=B 2023-03-20 00:55:45,064 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:45,064 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=C 2023-03-20 00:55:45,064 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:45,201 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:45,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:45,222 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:45,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273805219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:45,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:45,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273805222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:45,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:45,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273805323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:45,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:45,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273805324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:45,501 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/4c803743ec0c4760a51dd45f8f33f9a1 2023-03-20 00:55:45,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:45,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273805526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:45,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:45,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273805527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:45,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:45,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273805832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:45,834 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:45,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273805833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:45,915 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/1c9a01e3ee2044fa8fb51dc33533eb3c 2023-03-20 00:55:46,332 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/27fe31c5fdaf4e3d92a46f51ec4817d9 2023-03-20 00:55:46,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:46,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273806335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:46,338 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/4c803743ec0c4760a51dd45f8f33f9a1 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/4c803743ec0c4760a51dd45f8f33f9a1 2023-03-20 00:55:46,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:46,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273806339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:46,343 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/4c803743ec0c4760a51dd45f8f33f9a1, entries=150, sequenceid=579, filesize=11.9 K 2023-03-20 00:55:46,344 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/1c9a01e3ee2044fa8fb51dc33533eb3c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/1c9a01e3ee2044fa8fb51dc33533eb3c 2023-03-20 00:55:46,350 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/1c9a01e3ee2044fa8fb51dc33533eb3c, entries=150, sequenceid=579, filesize=11.9 K 2023-03-20 00:55:46,351 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/27fe31c5fdaf4e3d92a46f51ec4817d9 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/27fe31c5fdaf4e3d92a46f51ec4817d9 2023-03-20 00:55:46,356 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/27fe31c5fdaf4e3d92a46f51ec4817d9, entries=150, sequenceid=579, filesize=11.9 K 2023-03-20 00:55:46,357 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 1bfc994bd080bb1b8cbc4aef89c8cc98 in 1294ms, sequenceid=579, compaction requested=true 2023-03-20 00:55:46,357 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:46,357 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:46,357 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:46,358 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:46,358 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:46,358 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:46,358 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:46,359 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:46,359 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:46,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:46,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:46,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:46,359 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:46,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:46,359 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:46,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:46,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:46,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:46,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:46,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:46,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53cff5f6[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:46,361 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:46,361 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:46,362 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:46,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:46,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:46,362 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:46,362 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:46,362 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:46,362 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:46,362 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:46,363 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:46,363 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:46,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:46,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:46,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:46,363 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:46,363 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:46,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:46,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:46,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:46,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:46,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:46,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:46,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:46,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:46,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:46,368 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:46,368 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:46,368 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:46,368 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:46,368 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:46,369 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:46,369 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:46,370 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:46,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:46,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:46,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:46,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:46,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:46,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:46,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:46,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:46,371 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:46,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:46,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:46,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:46,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:46,372 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:46,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:46,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:46,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:46,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:46,374 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:46,374 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:46,374 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:46,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:46,374 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:46,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:46,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:46,374 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:46,374 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:46,374 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:46,374 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:46,374 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:46,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:46,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:46,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:46,374 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:46,375 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:46,375 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:46,474 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:46,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:46,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:46,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:46,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:46,478 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:46,478 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:46,478 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:46,478 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:46,478 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:46,478 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:46,479 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:46,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:46,479 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:46,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:46,479 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:46,479 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:46,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:46,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:46,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:46,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:46,480 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:46,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:46,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:46,480 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:46,480 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:46,480 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:46,482 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:46,482 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:46,482 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:46,482 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:46,482 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bfc994bd080bb1b8cbc4aef89c8cc98 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-20 00:55:46,482 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=A 2023-03-20 00:55:46,482 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:46,483 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=B 2023-03-20 00:55:46,483 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:46,483 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=C 2023-03-20 00:55:46,483 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:46,894 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=591 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/f4703c00a4174898bea12fa7a8c43565 2023-03-20 00:55:47,322 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=591 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/9ab88c4487aa49f59ec541cb7c27033e 2023-03-20 00:55:47,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:47,340 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:47,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:47,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273807371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,374 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:47,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273807372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:47,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273807474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:47,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273807475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,677 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:47,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:47,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273807676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273807676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,736 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=591 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/b9d5ee98cd9049ec911343f248d348f4 2023-03-20 00:55:47,742 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/f4703c00a4174898bea12fa7a8c43565 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/f4703c00a4174898bea12fa7a8c43565 2023-03-20 00:55:47,747 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/f4703c00a4174898bea12fa7a8c43565, entries=150, sequenceid=591, filesize=11.9 K 2023-03-20 00:55:47,748 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/9ab88c4487aa49f59ec541cb7c27033e as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/9ab88c4487aa49f59ec541cb7c27033e 2023-03-20 00:55:47,753 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/9ab88c4487aa49f59ec541cb7c27033e, entries=150, sequenceid=591, filesize=11.9 K 2023-03-20 00:55:47,754 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/b9d5ee98cd9049ec911343f248d348f4 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/b9d5ee98cd9049ec911343f248d348f4 2023-03-20 00:55:47,763 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/b9d5ee98cd9049ec911343f248d348f4, entries=150, sequenceid=591, filesize=11.9 K 2023-03-20 00:55:47,764 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 1bfc994bd080bb1b8cbc4aef89c8cc98 in 1281ms, sequenceid=591, compaction requested=true 2023-03-20 00:55:47,764 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:47,764 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:47,764 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:47,764 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:47,764 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,764 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:47,764 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:47,765 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:47,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:47,765 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:47,765 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:47,765 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:47,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:47,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:47,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:47,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:47,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:47,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b0a4a04[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:47,766 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:47,766 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:47,767 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:47,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:47,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:47,767 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,767 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:47,767 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:47,767 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:47,767 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:47,768 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,768 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:47,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:47,768 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:47,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:47,768 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:47,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:47,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:47,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:47,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:47,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:47,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:47,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:47,770 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:47,771 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:47,771 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:47,771 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:47,771 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:47,772 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:47,772 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:47,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:47,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:47,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:47,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:47,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:47,772 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:47,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:47,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:47,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:47,773 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:47,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:47,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:47,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:47,774 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:47,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:47,776 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:47,776 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:47,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:47,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:47,777 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,777 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:47,777 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:47,777 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:47,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:47,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:47,777 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,777 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:47,777 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:47,777 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:47,777 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:47,777 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:47,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:47,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:47,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:47,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:47,784 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1bfc994bd080bb1b8cbc4aef89c8cc98 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-20 00:55:47,785 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=A 2023-03-20 00:55:47,785 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:47,785 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=B 2023-03-20 00:55:47,785 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:47,785 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=C 2023-03-20 00:55:47,785 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:47,796 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=612 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/e7dcccd741604687b19eef4d42517bcd 2023-03-20 00:55:47,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:47,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37902 deadline: 1679273807797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:47,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37938 deadline: 1679273807797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:47,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37956 deadline: 1679273807800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,877 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:47,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:47,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:47,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:47,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:47,880 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:47,880 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:47,880 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:47,880 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:47,880 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:47,880 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:47,881 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:47,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:47,881 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:47,881 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,881 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:47,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:47,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:47,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:47,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:47,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:47,882 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:47,882 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:47,882 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:47,882 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:47,882 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:47,882 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:47,883 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:47,883 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:47,883 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:47,883 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:47,883 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:47,883 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:47,883 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:47,883 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,883 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:47,883 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:47,884 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,884 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:47,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:47,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:47,884 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:47,884 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:47,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:47,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:47,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:47,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:47,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:47,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a6f410[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:47,891 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:47,891 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:47,896 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:47,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:47,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:47,897 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:47,897 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:47,897 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:47,898 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:47,898 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:47,898 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:47,899 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,899 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,899 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:47,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:47,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:47,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:47,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:47,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:47,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:47,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:47,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:47,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:47,902 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:47,902 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:47,902 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:47,902 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:47,902 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:47,903 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:47,903 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:47,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:47,904 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:47,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:47,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:47,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:47,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:47,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:47,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:47,906 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,906 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:47,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:47,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37902 deadline: 1679273807904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,906 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:47,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37938 deadline: 1679273807904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,907 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,908 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:47,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37956 deadline: 1679273807907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,918 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,918 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:47,918 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:47,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:47,918 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:47,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:47,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:47,918 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:47,919 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:47,919 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:47,919 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:47,919 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:47,919 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:47,919 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:47,919 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:47,979 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:47,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:47,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273807978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:47,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273807979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,019 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:48,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:48,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:48,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:48,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:48,022 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:48,022 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:48,022 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:48,022 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:48,022 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:48,022 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:48,023 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:48,023 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:48,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:48,023 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,023 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:48,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:48,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:48,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:48,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:48,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:48,024 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:48,024 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:48,024 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:48,024 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:48,024 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:48,024 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:48,024 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:48,024 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:48,024 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:48,025 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:48,025 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:48,025 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:48,025 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:48,025 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,025 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:48,025 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:48,026 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,026 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:48,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:48,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:48,026 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:48,026 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:48,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:48,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:48,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:48,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:48,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:48,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32ed9660[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:48,027 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:48,028 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:48,028 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:48,028 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:48,028 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:48,029 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:48,029 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:48,029 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:48,030 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:48,030 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:48,030 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:48,031 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:48,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:48,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:48,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:48,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:48,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:48,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:48,034 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:48,034 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:48,034 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:48,034 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:48,034 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:48,035 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:48,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:48,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:48,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:48,035 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:48,036 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:48,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:48,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:48,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:48,036 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:48,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:48,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:48,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:48,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,039 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:48,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,040 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:48,040 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:48,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:48,040 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,041 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:48,041 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:48,041 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,041 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:48,041 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:48,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:48,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:48,041 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:48,041 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:48,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:48,041 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:48,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:48,041 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:48,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:48,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:48,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37902 deadline: 1679273808108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:48,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37938 deadline: 1679273808109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:48,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37956 deadline: 1679273808109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,141 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:48,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:48,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:48,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:48,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:48,146 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:48,146 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:48,146 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:48,146 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:48,147 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:48,147 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:48,147 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,147 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:48,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:48,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:48,148 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,148 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:48,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:48,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:48,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:48,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:48,149 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:48,153 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:48,153 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:48,153 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:48,153 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:48,153 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:48,153 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:48,153 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:48,153 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:48,153 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:48,153 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:48,153 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:48,153 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:48,153 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:48,153 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,153 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:48,153 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:48,159 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,159 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:48,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:48,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:48,159 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:48,159 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:48,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:48,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:48,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:48,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:48,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:48,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ea1a58e[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:48,160 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:48,160 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:48,179 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:48,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:48,179 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:48,180 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:48,180 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:48,180 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,180 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:48,182 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:48,182 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,182 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:48,182 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:48,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:48,184 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:48,184 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:48,184 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:48,184 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:48,184 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:48,192 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:48,192 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:48,192 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:48,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:48,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:48,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:48,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:48,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:48,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:48,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:48,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:48,193 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:48,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:48,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:48,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:48,195 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:48,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:48,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,200 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,201 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:48,201 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:48,201 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,201 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:48,201 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:48,201 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:48,201 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:48,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:48,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:48,201 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:48,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:48,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:48,201 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:48,201 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:48,201 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:48,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:48,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:48,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:48,245 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=612 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/e306f3efc5cb4078a1e080f1c93d379a 2023-03-20 00:55:48,256 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=612 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/32d26bd2c7a848e2b11db0ad1c55d5e4 2023-03-20 00:55:48,261 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/e7dcccd741604687b19eef4d42517bcd as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/e7dcccd741604687b19eef4d42517bcd 2023-03-20 00:55:48,267 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/e7dcccd741604687b19eef4d42517bcd, entries=200, sequenceid=612, filesize=14.3 K 2023-03-20 00:55:48,269 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/e306f3efc5cb4078a1e080f1c93d379a as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/e306f3efc5cb4078a1e080f1c93d379a 2023-03-20 00:55:48,276 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/e306f3efc5cb4078a1e080f1c93d379a, entries=150, sequenceid=612, filesize=11.9 K 2023-03-20 00:55:48,279 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/32d26bd2c7a848e2b11db0ad1c55d5e4 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/32d26bd2c7a848e2b11db0ad1c55d5e4 2023-03-20 00:55:48,289 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/32d26bd2c7a848e2b11db0ad1c55d5e4, entries=150, sequenceid=612, filesize=11.9 K 2023-03-20 00:55:48,292 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 1bfc994bd080bb1b8cbc4aef89c8cc98 in 507ms, sequenceid=612, compaction requested=true 2023-03-20 00:55:48,292 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:48,292 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:55:48,292 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:55:48,292 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-20 00:55:48,292 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-20 00:55:48,292 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:55:48,294 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73866 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-20 00:55:48,294 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 76306 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-20 00:55:48,294 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 1bfc994bd080bb1b8cbc4aef89c8cc98/B is initiating minor compaction (all files) 2023-03-20 00:55:48,294 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 1bfc994bd080bb1b8cbc4aef89c8cc98/A is initiating minor compaction (all files) 2023-03-20 00:55:48,294 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bfc994bd080bb1b8cbc4aef89c8cc98/B in TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:48,294 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bfc994bd080bb1b8cbc4aef89c8cc98/A in TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:48,294 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/106fdffcb1d845d5a7b76bf4b01a4ccf, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/4e9daa43cb10457293fa1e108674bbb4, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/d20fd632f17c4272b4de8e4953494440, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/1c9a01e3ee2044fa8fb51dc33533eb3c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/9ab88c4487aa49f59ec541cb7c27033e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/e306f3efc5cb4078a1e080f1c93d379a] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp, totalSize=72.1 K 2023-03-20 00:55:48,294 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/7273e3433837438fa3ea8d1dc05d2129, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/331b549410754af7bae1d19679be1e8d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/9b344171f5494d96922b03c6b8515c0c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/4c803743ec0c4760a51dd45f8f33f9a1, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/f4703c00a4174898bea12fa7a8c43565, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/e7dcccd741604687b19eef4d42517bcd] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp, totalSize=74.5 K 2023-03-20 00:55:48,295 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 106fdffcb1d845d5a7b76bf4b01a4ccf, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1679273742405 2023-03-20 00:55:48,295 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 7273e3433837438fa3ea8d1dc05d2129, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1679273742405 2023-03-20 00:55:48,295 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 4e9daa43cb10457293fa1e108674bbb4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1679273742776 2023-03-20 00:55:48,295 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 331b549410754af7bae1d19679be1e8d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1679273742776 2023-03-20 00:55:48,296 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting d20fd632f17c4272b4de8e4953494440, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=559, earliestPutTs=1679273743408 2023-03-20 00:55:48,296 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 9b344171f5494d96922b03c6b8515c0c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=559, earliestPutTs=1679273743408 2023-03-20 00:55:48,296 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 1c9a01e3ee2044fa8fb51dc33533eb3c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1679273744038 2023-03-20 00:55:48,296 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 4c803743ec0c4760a51dd45f8f33f9a1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1679273744038 2023-03-20 00:55:48,296 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 9ab88c4487aa49f59ec541cb7c27033e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=591, earliestPutTs=1679273745211 2023-03-20 00:55:48,297 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting f4703c00a4174898bea12fa7a8c43565, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=591, earliestPutTs=1679273745211 2023-03-20 00:55:48,297 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting e306f3efc5cb4078a1e080f1c93d379a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=612, earliestPutTs=1679273747358 2023-03-20 00:55:48,297 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting e7dcccd741604687b19eef4d42517bcd, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=612, earliestPutTs=1679273747358 2023-03-20 00:55:48,301 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:48,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:48,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:48,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:48,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:48,306 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:48,306 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:48,306 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:48,306 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:48,306 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:48,306 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:48,308 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:48,308 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:48,308 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,308 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:48,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:48,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:48,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:48,309 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:48,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:48,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:48,319 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:48,319 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:48,319 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:48,322 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 1bfc994bd080bb1b8cbc4aef89c8cc98#B#compaction#84 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:55:48,322 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:48,322 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:48,323 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:48,323 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:48,323 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bfc994bd080bb1b8cbc4aef89c8cc98 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-20 00:55:48,323 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=A 2023-03-20 00:55:48,323 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:48,323 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=B 2023-03-20 00:55:48,323 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:48,323 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=C 2023-03-20 00:55:48,324 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:48,324 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1bfc994bd080bb1b8cbc4aef89c8cc98#A#compaction#85 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:55:48,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:48,414 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:48,444 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:48,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679273748434, max timestamp=1679273748443Num uniques 859; 2023-03-20 00:55:48,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37938 deadline: 1679273808442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679273748434, max timestamp=1679273748443Num uniques 859; 2023-03-20 00:55:48,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 00:55:48,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 00:55:48,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:48,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37902 deadline: 1679273808463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:48,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37956 deadline: 1679273808467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,483 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:48,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273808480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,483 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:48,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273808480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,547 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:48,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37938 deadline: 1679273808546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,571 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:48,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37902 deadline: 1679273808570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:48,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37956 deadline: 1679273808572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,752 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:48,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37938 deadline: 1679273808749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,752 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=623 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/e4441371ac374507b331e2f7ae179ede 2023-03-20 00:55:48,772 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/adae2240e012406d9a76ca9e274bba1d as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/adae2240e012406d9a76ca9e274bba1d 2023-03-20 00:55:48,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:48,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37902 deadline: 1679273808774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,778 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 1bfc994bd080bb1b8cbc4aef89c8cc98/A of 1bfc994bd080bb1b8cbc4aef89c8cc98 into adae2240e012406d9a76ca9e274bba1d(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:55:48,778 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:48,778 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98., storeName=1bfc994bd080bb1b8cbc4aef89c8cc98/A, priority=10, startTime=1679273748292; duration=0sec 2023-03-20 00:55:48,778 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:55:48,778 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-20 00:55:48,780 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73866 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-20 00:55:48,781 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 1bfc994bd080bb1b8cbc4aef89c8cc98/C is initiating minor compaction (all files) 2023-03-20 00:55:48,781 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bfc994bd080bb1b8cbc4aef89c8cc98/C in TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:48,781 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/6cdd3bfcbb1c473fbe6b81b03b427cad, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/99ab2e310fe24a71a493576e706e685c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/88948025020d4bada6adb03b0a4e120d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/27fe31c5fdaf4e3d92a46f51ec4817d9, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/b9d5ee98cd9049ec911343f248d348f4, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/32d26bd2c7a848e2b11db0ad1c55d5e4] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp, totalSize=72.1 K 2023-03-20 00:55:48,781 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 6cdd3bfcbb1c473fbe6b81b03b427cad, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1679273742405 2023-03-20 00:55:48,781 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 99ab2e310fe24a71a493576e706e685c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1679273742776 2023-03-20 00:55:48,782 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 88948025020d4bada6adb03b0a4e120d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=559, earliestPutTs=1679273743408 2023-03-20 00:55:48,782 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 27fe31c5fdaf4e3d92a46f51ec4817d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1679273744038 2023-03-20 00:55:48,783 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting b9d5ee98cd9049ec911343f248d348f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=591, earliestPutTs=1679273745211 2023-03-20 00:55:48,783 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 32d26bd2c7a848e2b11db0ad1c55d5e4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=612, earliestPutTs=1679273747358 2023-03-20 00:55:48,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:48,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37956 deadline: 1679273808779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:48,802 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/e31408ce2b274cc2af7ba44b0acd9ddb as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/e31408ce2b274cc2af7ba44b0acd9ddb 2023-03-20 00:55:48,805 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1bfc994bd080bb1b8cbc4aef89c8cc98#C#compaction#88 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:55:48,808 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 1bfc994bd080bb1b8cbc4aef89c8cc98/B of 1bfc994bd080bb1b8cbc4aef89c8cc98 into e31408ce2b274cc2af7ba44b0acd9ddb(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:55:48,808 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:48,808 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98., storeName=1bfc994bd080bb1b8cbc4aef89c8cc98/B, priority=10, startTime=1679273748292; duration=0sec 2023-03-20 00:55:48,809 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:55:48,831 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/a4fa96b75c8c459e89b3458f9b0ac74b as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/a4fa96b75c8c459e89b3458f9b0ac74b 2023-03-20 00:55:48,838 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 1bfc994bd080bb1b8cbc4aef89c8cc98/C of 1bfc994bd080bb1b8cbc4aef89c8cc98 into a4fa96b75c8c459e89b3458f9b0ac74b(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:55:48,838 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:48,838 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98., storeName=1bfc994bd080bb1b8cbc4aef89c8cc98/C, priority=10, startTime=1679273748292; duration=0sec 2023-03-20 00:55:48,838 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:55:49,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:49,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37938 deadline: 1679273809054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:49,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:49,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37902 deadline: 1679273809079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:49,088 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:49,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37956 deadline: 1679273809087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:49,198 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=623 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/a1b3cd032d57406db364eaab182cc974 2023-03-20 00:55:49,209 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=623 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/a770bc4695e040588b23d0e8c983981a 2023-03-20 00:55:49,215 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/e4441371ac374507b331e2f7ae179ede as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/e4441371ac374507b331e2f7ae179ede 2023-03-20 00:55:49,219 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/e4441371ac374507b331e2f7ae179ede, entries=150, sequenceid=623, filesize=11.9 K 2023-03-20 00:55:49,220 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/a1b3cd032d57406db364eaab182cc974 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/a1b3cd032d57406db364eaab182cc974 2023-03-20 00:55:49,225 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/a1b3cd032d57406db364eaab182cc974, entries=150, sequenceid=623, filesize=11.9 K 2023-03-20 00:55:49,226 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/a770bc4695e040588b23d0e8c983981a as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/a770bc4695e040588b23d0e8c983981a 2023-03-20 00:55:49,232 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/a770bc4695e040588b23d0e8c983981a, entries=150, sequenceid=623, filesize=11.9 K 2023-03-20 00:55:49,233 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 1bfc994bd080bb1b8cbc4aef89c8cc98 in 910ms, sequenceid=623, compaction requested=false 2023-03-20 00:55:49,233 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:49,233 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:49,233 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:49,233 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:49,233 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:49,235 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:49,236 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:49,237 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:49,237 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:49,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:49,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:49,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:49,238 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:49,238 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:49,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:49,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:49,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:49,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:49,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:49,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:49,239 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:49,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7628634[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:49,239 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:49,240 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:49,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:49,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:49,240 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:49,240 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:49,240 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:49,240 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:49,240 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:49,241 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:49,241 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:49,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:49,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:49,241 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:49,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:49,241 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:49,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:49,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:49,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:49,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:49,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:49,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:49,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:49,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:49,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:49,248 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:49,249 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:49,249 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:49,249 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:49,249 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:49,261 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:49,261 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:49,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:49,261 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:49,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:49,262 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:49,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:49,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:49,263 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:49,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:49,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:49,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:49,270 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:49,270 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:49,271 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:49,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:49,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:49,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:49,271 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:49,271 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:49,271 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:49,271 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:49,271 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:49,271 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:49,271 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:49,271 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:49,271 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:49,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:49,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:49,371 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:49,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:49,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:49,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:49,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:49,375 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:49,375 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:49,375 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:49,375 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:49,375 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:49,376 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:49,376 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:49,376 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:49,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:49,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:49,377 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:49,377 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:49,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:49,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:49,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:49,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:49,377 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:49,378 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:49,378 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:49,378 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:49,378 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:49,378 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:49,378 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:49,378 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:49,378 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:49,378 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. started... 2023-03-20 00:55:49,378 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bfc994bd080bb1b8cbc4aef89c8cc98 3/3 column families, dataSize=127.47 KB heapSize=368.47 KB 2023-03-20 00:55:49,378 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=A 2023-03-20 00:55:49,379 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:49,379 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=B 2023-03-20 00:55:49,379 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:49,379 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=C 2023-03-20 00:55:49,379 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:49,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:49,488 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. as already flushing 2023-03-20 00:55:49,497 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:49,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273809495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:49,498 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:49,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273809497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:49,558 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:49,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37938 deadline: 1679273809556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:49,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:49,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37902 deadline: 1679273809585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:49,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:49,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37956 deadline: 1679273809591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:49,599 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:49,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273809598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:49,600 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:49,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273809599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:49,786 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=657 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/3e642b2ab3704e0b9687e1090201a4ca 2023-03-20 00:55:49,800 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=657 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/f6a77fe47bb347c7bade58ad0701633d 2023-03-20 00:55:49,801 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:49,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273809801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:49,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:49,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273809802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:50,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:50,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37918 deadline: 1679273810104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:50,106 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:50,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:37946 deadline: 1679273810104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:50,181 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x655398b8 to 127.0.0.1:60547 2023-03-20 00:55:50,181 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:55:50,181 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x776686ab to 127.0.0.1:60547 2023-03-20 00:55:50,181 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:55:50,185 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5fe050a3 to 127.0.0.1:60547 2023-03-20 00:55:50,185 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:55:50,186 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x13112432 to 127.0.0.1:60547 2023-03-20 00:55:50,187 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:55:50,215 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=62.62 KB at sequenceid=657 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/32fe4bc6d9674629b080d8d2af37cc8e 2023-03-20 00:55:50,219 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/3e642b2ab3704e0b9687e1090201a4ca as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/3e642b2ab3704e0b9687e1090201a4ca 2023-03-20 00:55:50,224 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/3e642b2ab3704e0b9687e1090201a4ca, entries=150, sequenceid=657, filesize=11.9 K 2023-03-20 00:55:50,225 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/f6a77fe47bb347c7bade58ad0701633d as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/f6a77fe47bb347c7bade58ad0701633d 2023-03-20 00:55:50,230 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/f6a77fe47bb347c7bade58ad0701633d, entries=150, sequenceid=657, filesize=11.9 K 2023-03-20 00:55:50,231 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/32fe4bc6d9674629b080d8d2af37cc8e as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/32fe4bc6d9674629b080d8d2af37cc8e 2023-03-20 00:55:50,235 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/32fe4bc6d9674629b080d8d2af37cc8e, entries=150, sequenceid=657, filesize=11.9 K 2023-03-20 00:55:50,236 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~368.42 KB/377264, currentSize=53.67 KB/54960 for 1bfc994bd080bb1b8cbc4aef89c8cc98 in 858ms, sequenceid=657, compaction requested=true 2023-03-20 00:55:50,236 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:50,236 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:50,236 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:50,236 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:50,236 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:50,236 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:50,236 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:50,237 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:50,237 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:50,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:50,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:50,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:50,237 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:50,237 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:50,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:50,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:50,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:50,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:50,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:50,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:50,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34d7f4bc[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:50,239 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:50,239 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:50,239 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:50,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:50,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:50,239 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:50,239 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:50,239 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:50,239 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:50,239 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:50,240 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:50,240 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:50,240 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:50,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:50,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:50,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:50,240 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:50,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:50,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:50,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:50,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:50,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:50,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:50,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:50,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:50,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:50,242 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:50,242 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:50,242 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:50,242 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:50,242 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:50,244 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:50,244 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:50,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:50,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:50,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:50,244 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:50,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:50,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:50,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:50,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:50,244 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:50,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:50,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:50,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:50,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:50,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:50,245 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:50,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:50,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:50,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:50,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:50,246 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:50,246 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:50,246 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:50,247 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:50,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:50,247 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:50,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:50,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:50,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:50,247 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:50,247 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:50,247 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:50,247 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:50,247 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:50,247 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:50,247 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:50,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:50,347 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:50,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:50,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:50,564 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1bfc994bd080bb1b8cbc4aef89c8cc98 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-20 00:55:50,564 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=A 2023-03-20 00:55:50,564 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x20e4e4f3 to 127.0.0.1:60547 2023-03-20 00:55:50,564 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:50,565 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:55:50,565 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=B 2023-03-20 00:55:50,565 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:50,565 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=C 2023-03-20 00:55:50,566 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:50,598 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6362e5a0 to 127.0.0.1:60547 2023-03-20 00:55:50,598 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:55:50,600 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x142c8aea to 127.0.0.1:60547 2023-03-20 00:55:50,600 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:55:50,609 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x046b5e40 to 127.0.0.1:60547 2023-03-20 00:55:50,609 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:55:50,610 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1f537ba4 to 127.0.0.1:60547 2023-03-20 00:55:50,610 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:55:50,610 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-20 00:55:50,610 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(394): wrote 105 2023-03-20 00:55:50,610 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(394): wrote 86 2023-03-20 00:55:50,610 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(394): wrote 91 2023-03-20 00:55:50,610 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(394): wrote 151 2023-03-20 00:55:50,610 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(394): wrote 143 2023-03-20 00:55:50,610 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-20 00:55:50,610 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(398): read 4575 2023-03-20 00:55:50,610 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(398): read 4538 2023-03-20 00:55:50,610 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-20 00:55:50,610 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(402): scanned 2025 2023-03-20 00:55:50,610 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(403): verified 6074 rows 2023-03-20 00:55:50,611 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(402): scanned 1997 2023-03-20 00:55:50,611 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(403): verified 5987 rows 2023-03-20 00:55:50,611 INFO [Listener at localhost.localdomain/43361] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-03-20 00:55:50,611 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x458f9a9e to 127.0.0.1:60547 2023-03-20 00:55:50,611 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:55:50,615 INFO [Listener at localhost.localdomain/43361] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-20 00:55:50,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-03-20 00:55:50,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-20 00:55:50,630 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679273750630"}]},"ts":"1679273750630"} 2023-03-20 00:55:50,631 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-20 00:55:50,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-20 00:55:50,632 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-20 00:55:50,634 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1bfc994bd080bb1b8cbc4aef89c8cc98, UNASSIGN}] 2023-03-20 00:55:50,636 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1bfc994bd080bb1b8cbc4aef89c8cc98, UNASSIGN 2023-03-20 00:55:50,637 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=1bfc994bd080bb1b8cbc4aef89c8cc98, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:50,637 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273750637"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679273750637"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679273750637"}]},"ts":"1679273750637"} 2023-03-20 00:55:50,638 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199}] 2023-03-20 00:55:50,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-20 00:55:50,797 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:50,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1bfc994bd080bb1b8cbc4aef89c8cc98, disabling compactions & flushes 2023-03-20 00:55:50,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:50,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-20 00:55:50,979 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=669 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/aa213358db7843f6b73b1f787cf41054 2023-03-20 00:55:51,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-20 00:55:51,400 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=669 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/57805f0de05c40319cbc4dd42bad69cc 2023-03-20 00:55:51,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-20 00:55:51,819 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=669 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/3cc69dd4dc5c41b38e0e42e50d55a5f7 2023-03-20 00:55:51,827 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/aa213358db7843f6b73b1f787cf41054 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/aa213358db7843f6b73b1f787cf41054 2023-03-20 00:55:51,834 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/aa213358db7843f6b73b1f787cf41054, entries=150, sequenceid=669, filesize=11.9 K 2023-03-20 00:55:51,835 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/57805f0de05c40319cbc4dd42bad69cc as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/57805f0de05c40319cbc4dd42bad69cc 2023-03-20 00:55:51,841 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/57805f0de05c40319cbc4dd42bad69cc, entries=150, sequenceid=669, filesize=11.9 K 2023-03-20 00:55:51,842 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/3cc69dd4dc5c41b38e0e42e50d55a5f7 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/3cc69dd4dc5c41b38e0e42e50d55a5f7 2023-03-20 00:55:51,848 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/3cc69dd4dc5c41b38e0e42e50d55a5f7, entries=150, sequenceid=669, filesize=11.9 K 2023-03-20 00:55:51,849 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=26.84 KB/27480 for 1bfc994bd080bb1b8cbc4aef89c8cc98 in 1285ms, sequenceid=669, compaction requested=true 2023-03-20 00:55:51,849 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:51,850 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:51,850 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:55:51,850 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. because compaction request was cancelled 2023-03-20 00:55:51,850 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. because compaction request was cancelled 2023-03-20 00:55:51,850 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:55:51,850 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:51,850 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. because compaction request was cancelled 2023-03-20 00:55:51,850 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:55:51,850 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. after waiting 0 ms 2023-03-20 00:55:51,850 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:51,850 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 1bfc994bd080bb1b8cbc4aef89c8cc98 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-20 00:55:51,850 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=A 2023-03-20 00:55:51,850 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:51,850 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=B 2023-03-20 00:55:51,851 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:51,851 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bfc994bd080bb1b8cbc4aef89c8cc98, store=C 2023-03-20 00:55:51,851 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:52,263 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=676 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/94a3b82419a74025ba21398f9b90340a 2023-03-20 00:55:52,683 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=676 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/f960f12aea5e4c8481a72242073dad37 2023-03-20 00:55:52,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-20 00:55:53,104 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=676 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/8f7d019524bb4f7eb8dbc9e3b768b08c 2023-03-20 00:55:53,116 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/A/94a3b82419a74025ba21398f9b90340a as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/94a3b82419a74025ba21398f9b90340a 2023-03-20 00:55:53,122 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/94a3b82419a74025ba21398f9b90340a, entries=150, sequenceid=676, filesize=11.9 K 2023-03-20 00:55:53,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/B/f960f12aea5e4c8481a72242073dad37 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/f960f12aea5e4c8481a72242073dad37 2023-03-20 00:55:53,129 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/f960f12aea5e4c8481a72242073dad37, entries=150, sequenceid=676, filesize=11.9 K 2023-03-20 00:55:53,130 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/.tmp/C/8f7d019524bb4f7eb8dbc9e3b768b08c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/8f7d019524bb4f7eb8dbc9e3b768b08c 2023-03-20 00:55:53,137 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/8f7d019524bb4f7eb8dbc9e3b768b08c, entries=150, sequenceid=676, filesize=11.9 K 2023-03-20 00:55:53,138 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 1bfc994bd080bb1b8cbc4aef89c8cc98 in 1288ms, sequenceid=676, compaction requested=true 2023-03-20 00:55:53,165 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/eea2a04a552a4392935fc730d34dde0b, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/8d333f99151a43c2901b7c4665b8ea42, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/2cfe9b13638244fe80adc14cfc7c5e53, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/50f1ef0abcf3450c82a0303f781fa7f8, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/0b55f9469a7343cfbfa8e0b89fcc37f9, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/a16bedf52d5949e5b9fd61eadaad3b08, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/b1fd3ce5b70a458cb9e00a3c3d5e8d26, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/3df40fd94e3042ee86071d652342991e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/5aabcf44d83140bd943c5c4bed40cf52, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/a1a08a26289243bbaa20ca0fecaf3ed8, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/29529002dd4e497e9a959a58e65929be, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/6f2067f14d784cb198c1a1a7e8ce3e5b, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/c7e230ae4b09455784c5f2b1197f4d8f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/aa138909118345298b23c635e9f1522d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/4f5ee2a2b7bf449fb25e69a5e377deed, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/94942bd149a94d3f965897873a7eeaa0, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/c7e5d32c4f5c441f8f65a2a5d6bfabe7, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/de0967b2dbb3414d88e7a693227785be, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/e1cc1a9deca74ed182ff8db5f49136be, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/c4151e07b2a54a38b9fd40fe2f9b3c2c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/7fefe4cd442f4027b05d32a70edd9b0c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/861359865da649c1a36ef859679b177e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/7273e3433837438fa3ea8d1dc05d2129, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/331b549410754af7bae1d19679be1e8d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/9b344171f5494d96922b03c6b8515c0c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/4c803743ec0c4760a51dd45f8f33f9a1, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/f4703c00a4174898bea12fa7a8c43565, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/e7dcccd741604687b19eef4d42517bcd] to archive 2023-03-20 00:55:53,166 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 00:55:53,172 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/eea2a04a552a4392935fc730d34dde0b to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/eea2a04a552a4392935fc730d34dde0b 2023-03-20 00:55:53,173 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/8d333f99151a43c2901b7c4665b8ea42 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/8d333f99151a43c2901b7c4665b8ea42 2023-03-20 00:55:53,175 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/2cfe9b13638244fe80adc14cfc7c5e53 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/2cfe9b13638244fe80adc14cfc7c5e53 2023-03-20 00:55:53,176 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/50f1ef0abcf3450c82a0303f781fa7f8 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/50f1ef0abcf3450c82a0303f781fa7f8 2023-03-20 00:55:53,178 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/0b55f9469a7343cfbfa8e0b89fcc37f9 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/0b55f9469a7343cfbfa8e0b89fcc37f9 2023-03-20 00:55:53,180 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/a16bedf52d5949e5b9fd61eadaad3b08 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/a16bedf52d5949e5b9fd61eadaad3b08 2023-03-20 00:55:53,181 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/b1fd3ce5b70a458cb9e00a3c3d5e8d26 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/b1fd3ce5b70a458cb9e00a3c3d5e8d26 2023-03-20 00:55:53,182 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/3df40fd94e3042ee86071d652342991e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/3df40fd94e3042ee86071d652342991e 2023-03-20 00:55:53,183 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/5aabcf44d83140bd943c5c4bed40cf52 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/5aabcf44d83140bd943c5c4bed40cf52 2023-03-20 00:55:53,184 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/a1a08a26289243bbaa20ca0fecaf3ed8 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/a1a08a26289243bbaa20ca0fecaf3ed8 2023-03-20 00:55:53,185 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/29529002dd4e497e9a959a58e65929be to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/29529002dd4e497e9a959a58e65929be 2023-03-20 00:55:53,186 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/6f2067f14d784cb198c1a1a7e8ce3e5b to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/6f2067f14d784cb198c1a1a7e8ce3e5b 2023-03-20 00:55:53,188 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/c7e230ae4b09455784c5f2b1197f4d8f to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/c7e230ae4b09455784c5f2b1197f4d8f 2023-03-20 00:55:53,189 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/aa138909118345298b23c635e9f1522d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/aa138909118345298b23c635e9f1522d 2023-03-20 00:55:53,189 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/4f5ee2a2b7bf449fb25e69a5e377deed to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/4f5ee2a2b7bf449fb25e69a5e377deed 2023-03-20 00:55:53,191 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/94942bd149a94d3f965897873a7eeaa0 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/94942bd149a94d3f965897873a7eeaa0 2023-03-20 00:55:53,193 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/c7e5d32c4f5c441f8f65a2a5d6bfabe7 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/c7e5d32c4f5c441f8f65a2a5d6bfabe7 2023-03-20 00:55:53,194 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/de0967b2dbb3414d88e7a693227785be to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/de0967b2dbb3414d88e7a693227785be 2023-03-20 00:55:53,195 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/e1cc1a9deca74ed182ff8db5f49136be to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/e1cc1a9deca74ed182ff8db5f49136be 2023-03-20 00:55:53,197 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/c4151e07b2a54a38b9fd40fe2f9b3c2c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/c4151e07b2a54a38b9fd40fe2f9b3c2c 2023-03-20 00:55:53,198 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/7fefe4cd442f4027b05d32a70edd9b0c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/7fefe4cd442f4027b05d32a70edd9b0c 2023-03-20 00:55:53,199 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/861359865da649c1a36ef859679b177e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/861359865da649c1a36ef859679b177e 2023-03-20 00:55:53,200 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/7273e3433837438fa3ea8d1dc05d2129 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/7273e3433837438fa3ea8d1dc05d2129 2023-03-20 00:55:53,201 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/331b549410754af7bae1d19679be1e8d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/331b549410754af7bae1d19679be1e8d 2023-03-20 00:55:53,202 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/9b344171f5494d96922b03c6b8515c0c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/9b344171f5494d96922b03c6b8515c0c 2023-03-20 00:55:53,203 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/4c803743ec0c4760a51dd45f8f33f9a1 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/4c803743ec0c4760a51dd45f8f33f9a1 2023-03-20 00:55:53,204 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/f4703c00a4174898bea12fa7a8c43565 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/f4703c00a4174898bea12fa7a8c43565 2023-03-20 00:55:53,205 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/e7dcccd741604687b19eef4d42517bcd to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/e7dcccd741604687b19eef4d42517bcd 2023-03-20 00:55:53,227 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/0426b3b87122418ab15e6a2d25434e33, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/0285bbe78f1142dd88d926aa91c86070, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/347e198bc2284d06916b6a674d49c885, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/6b4c59e3f0e843e1b3a1308f31a923b7, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/fe63a3a3ddd042099d41fb6d367bc674, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/0146222707c242238955d67ab2f3ea96, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/aeab3abfa1874b35bc3239c7e015c4bd, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/a044c4f4d79d4ebdacc14e00f6db10fb, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/34ea5ab100ef432389f26d243ad17f50, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/dfa8ca2ac4254894ae06dfb291d68a88, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/b5099528c0ca480aba53d1e3c17ca99c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/fe5d86fd78a74f21a643035560285986, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/dc645a9789a845ebbfacf846bd3f5e8c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/0d48ea45d4e447c193cb3c322ee72eb9, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/6b4da5d1bde249e383c84a314e843b72, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/2aab22b4007b46e5921ba0708fcfd3a8, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/43460b6977bb40b5a7e065910721a998, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/410016499759442c99ad10333e600121, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/105d742d161643a9b5e0ec40345df98f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/0d8f92ddd86f4425a736d80c425ac153, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/15ec207d262d437bba2cf616e8e5b1cd, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/106fdffcb1d845d5a7b76bf4b01a4ccf, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/ee0877aad135400ea931cc7b17d114cf, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/4e9daa43cb10457293fa1e108674bbb4, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/d20fd632f17c4272b4de8e4953494440, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/1c9a01e3ee2044fa8fb51dc33533eb3c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/9ab88c4487aa49f59ec541cb7c27033e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/e306f3efc5cb4078a1e080f1c93d379a] to archive 2023-03-20 00:55:53,229 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 00:55:53,230 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/0426b3b87122418ab15e6a2d25434e33 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/0426b3b87122418ab15e6a2d25434e33 2023-03-20 00:55:53,232 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/0285bbe78f1142dd88d926aa91c86070 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/0285bbe78f1142dd88d926aa91c86070 2023-03-20 00:55:53,233 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/347e198bc2284d06916b6a674d49c885 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/347e198bc2284d06916b6a674d49c885 2023-03-20 00:55:53,234 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/6b4c59e3f0e843e1b3a1308f31a923b7 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/6b4c59e3f0e843e1b3a1308f31a923b7 2023-03-20 00:55:53,235 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/fe63a3a3ddd042099d41fb6d367bc674 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/fe63a3a3ddd042099d41fb6d367bc674 2023-03-20 00:55:53,237 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/0146222707c242238955d67ab2f3ea96 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/0146222707c242238955d67ab2f3ea96 2023-03-20 00:55:53,238 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/aeab3abfa1874b35bc3239c7e015c4bd to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/aeab3abfa1874b35bc3239c7e015c4bd 2023-03-20 00:55:53,239 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/a044c4f4d79d4ebdacc14e00f6db10fb to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/a044c4f4d79d4ebdacc14e00f6db10fb 2023-03-20 00:55:53,240 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/34ea5ab100ef432389f26d243ad17f50 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/34ea5ab100ef432389f26d243ad17f50 2023-03-20 00:55:53,241 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/dfa8ca2ac4254894ae06dfb291d68a88 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/dfa8ca2ac4254894ae06dfb291d68a88 2023-03-20 00:55:53,242 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/b5099528c0ca480aba53d1e3c17ca99c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/b5099528c0ca480aba53d1e3c17ca99c 2023-03-20 00:55:53,243 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/fe5d86fd78a74f21a643035560285986 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/fe5d86fd78a74f21a643035560285986 2023-03-20 00:55:53,244 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/dc645a9789a845ebbfacf846bd3f5e8c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/dc645a9789a845ebbfacf846bd3f5e8c 2023-03-20 00:55:53,245 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/0d48ea45d4e447c193cb3c322ee72eb9 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/0d48ea45d4e447c193cb3c322ee72eb9 2023-03-20 00:55:53,246 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/6b4da5d1bde249e383c84a314e843b72 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/6b4da5d1bde249e383c84a314e843b72 2023-03-20 00:55:53,247 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/2aab22b4007b46e5921ba0708fcfd3a8 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/2aab22b4007b46e5921ba0708fcfd3a8 2023-03-20 00:55:53,248 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/43460b6977bb40b5a7e065910721a998 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/43460b6977bb40b5a7e065910721a998 2023-03-20 00:55:53,249 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/410016499759442c99ad10333e600121 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/410016499759442c99ad10333e600121 2023-03-20 00:55:53,250 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/105d742d161643a9b5e0ec40345df98f to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/105d742d161643a9b5e0ec40345df98f 2023-03-20 00:55:53,251 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/0d8f92ddd86f4425a736d80c425ac153 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/0d8f92ddd86f4425a736d80c425ac153 2023-03-20 00:55:53,252 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/15ec207d262d437bba2cf616e8e5b1cd to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/15ec207d262d437bba2cf616e8e5b1cd 2023-03-20 00:55:53,253 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/106fdffcb1d845d5a7b76bf4b01a4ccf to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/106fdffcb1d845d5a7b76bf4b01a4ccf 2023-03-20 00:55:53,255 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/ee0877aad135400ea931cc7b17d114cf to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/ee0877aad135400ea931cc7b17d114cf 2023-03-20 00:55:53,256 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/4e9daa43cb10457293fa1e108674bbb4 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/4e9daa43cb10457293fa1e108674bbb4 2023-03-20 00:55:53,258 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/d20fd632f17c4272b4de8e4953494440 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/d20fd632f17c4272b4de8e4953494440 2023-03-20 00:55:53,259 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/1c9a01e3ee2044fa8fb51dc33533eb3c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/1c9a01e3ee2044fa8fb51dc33533eb3c 2023-03-20 00:55:53,260 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/9ab88c4487aa49f59ec541cb7c27033e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/9ab88c4487aa49f59ec541cb7c27033e 2023-03-20 00:55:53,262 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/e306f3efc5cb4078a1e080f1c93d379a to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/e306f3efc5cb4078a1e080f1c93d379a 2023-03-20 00:55:53,266 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/f90ebde976224da8b38697ea810d80ea, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/ac6f26105a1f46caad27266964fd5b20, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/ce98b1cbce9a44878321d2521abea0d2, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/ba4ce765668d4051b8a1b62f3608f8f6, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/65630d250fff487ca88bbf272c53662e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/03d073d5708c48b9ac8404b6a27e310e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/15525c7551e2466d81d50c5cb82bd03c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/bccd87ae84bc4812b5092589c3ae7c21, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/93d6ccb109e846fdb81ab29f37053252, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/a9db4a5e73ae48919e7b3f395d6a862f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/aa41caeb809640c9839f400a70eb33c1, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/8e575f5f53dc492782e94551a99ef4ce, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/5a71ef4d830a48d9b73029c2f6f5501a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/fe2c78ad9d9d426e98f5323ba9bfe97d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/7b2193d932cb4f0c8e68877151620d7f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/58b3869641fe44fe9ed04c58a3ac0e99, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/a454bd679290463a85e2f320753deca1, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/8ad87668756b4df79d3bdf98c265902a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/2689a84dcc1c41a8affa864d3776e66d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/5be3121dcef34621ac78d50eba01106b, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/9516b841e1064e89bc4bf56db525d658, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/6cdd3bfcbb1c473fbe6b81b03b427cad, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/129ef095c05f4166be5cf23a02c40b1e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/99ab2e310fe24a71a493576e706e685c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/88948025020d4bada6adb03b0a4e120d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/27fe31c5fdaf4e3d92a46f51ec4817d9, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/b9d5ee98cd9049ec911343f248d348f4, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/32d26bd2c7a848e2b11db0ad1c55d5e4] to archive 2023-03-20 00:55:53,267 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 00:55:53,268 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/f90ebde976224da8b38697ea810d80ea to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/f90ebde976224da8b38697ea810d80ea 2023-03-20 00:55:53,270 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/ac6f26105a1f46caad27266964fd5b20 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/ac6f26105a1f46caad27266964fd5b20 2023-03-20 00:55:53,271 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/ce98b1cbce9a44878321d2521abea0d2 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/ce98b1cbce9a44878321d2521abea0d2 2023-03-20 00:55:53,272 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/ba4ce765668d4051b8a1b62f3608f8f6 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/ba4ce765668d4051b8a1b62f3608f8f6 2023-03-20 00:55:53,273 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/65630d250fff487ca88bbf272c53662e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/65630d250fff487ca88bbf272c53662e 2023-03-20 00:55:53,275 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/03d073d5708c48b9ac8404b6a27e310e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/03d073d5708c48b9ac8404b6a27e310e 2023-03-20 00:55:53,276 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/15525c7551e2466d81d50c5cb82bd03c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/15525c7551e2466d81d50c5cb82bd03c 2023-03-20 00:55:53,277 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/bccd87ae84bc4812b5092589c3ae7c21 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/bccd87ae84bc4812b5092589c3ae7c21 2023-03-20 00:55:53,279 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/93d6ccb109e846fdb81ab29f37053252 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/93d6ccb109e846fdb81ab29f37053252 2023-03-20 00:55:53,280 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/a9db4a5e73ae48919e7b3f395d6a862f to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/a9db4a5e73ae48919e7b3f395d6a862f 2023-03-20 00:55:53,281 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/aa41caeb809640c9839f400a70eb33c1 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/aa41caeb809640c9839f400a70eb33c1 2023-03-20 00:55:53,283 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/8e575f5f53dc492782e94551a99ef4ce to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/8e575f5f53dc492782e94551a99ef4ce 2023-03-20 00:55:53,284 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/5a71ef4d830a48d9b73029c2f6f5501a to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/5a71ef4d830a48d9b73029c2f6f5501a 2023-03-20 00:55:53,285 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/fe2c78ad9d9d426e98f5323ba9bfe97d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/fe2c78ad9d9d426e98f5323ba9bfe97d 2023-03-20 00:55:53,287 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/7b2193d932cb4f0c8e68877151620d7f to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/7b2193d932cb4f0c8e68877151620d7f 2023-03-20 00:55:53,288 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/58b3869641fe44fe9ed04c58a3ac0e99 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/58b3869641fe44fe9ed04c58a3ac0e99 2023-03-20 00:55:53,289 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/a454bd679290463a85e2f320753deca1 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/a454bd679290463a85e2f320753deca1 2023-03-20 00:55:53,290 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/8ad87668756b4df79d3bdf98c265902a to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/8ad87668756b4df79d3bdf98c265902a 2023-03-20 00:55:53,292 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/2689a84dcc1c41a8affa864d3776e66d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/2689a84dcc1c41a8affa864d3776e66d 2023-03-20 00:55:53,293 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/5be3121dcef34621ac78d50eba01106b to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/5be3121dcef34621ac78d50eba01106b 2023-03-20 00:55:53,294 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/9516b841e1064e89bc4bf56db525d658 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/9516b841e1064e89bc4bf56db525d658 2023-03-20 00:55:53,296 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/6cdd3bfcbb1c473fbe6b81b03b427cad to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/6cdd3bfcbb1c473fbe6b81b03b427cad 2023-03-20 00:55:53,297 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/129ef095c05f4166be5cf23a02c40b1e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/129ef095c05f4166be5cf23a02c40b1e 2023-03-20 00:55:53,298 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/99ab2e310fe24a71a493576e706e685c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/99ab2e310fe24a71a493576e706e685c 2023-03-20 00:55:53,299 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/88948025020d4bada6adb03b0a4e120d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/88948025020d4bada6adb03b0a4e120d 2023-03-20 00:55:53,301 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/27fe31c5fdaf4e3d92a46f51ec4817d9 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/27fe31c5fdaf4e3d92a46f51ec4817d9 2023-03-20 00:55:53,302 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/b9d5ee98cd9049ec911343f248d348f4 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/b9d5ee98cd9049ec911343f248d348f4 2023-03-20 00:55:53,304 DEBUG [StoreCloser-TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/32d26bd2c7a848e2b11db0ad1c55d5e4 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/32d26bd2c7a848e2b11db0ad1c55d5e4 2023-03-20 00:55:53,309 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/recovered.edits/679.seqid, newMaxSeqId=679, maxSeqId=1 2023-03-20 00:55:53,310 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98. 2023-03-20 00:55:53,310 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1bfc994bd080bb1b8cbc4aef89c8cc98: 2023-03-20 00:55:53,312 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:53,313 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=1bfc994bd080bb1b8cbc4aef89c8cc98, regionState=CLOSED 2023-03-20 00:55:53,313 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273753313"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679273753313"}]},"ts":"1679273753313"} 2023-03-20 00:55:53,317 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-03-20 00:55:53,317 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 1bfc994bd080bb1b8cbc4aef89c8cc98, server=jenkins-hbase20.apache.org,34553,1679273725199 in 2.6760 sec 2023-03-20 00:55:53,318 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-03-20 00:55:53,318 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1bfc994bd080bb1b8cbc4aef89c8cc98, UNASSIGN in 2.6830 sec 2023-03-20 00:55:53,319 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679273753319"}]},"ts":"1679273753319"} 2023-03-20 00:55:53,320 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-20 00:55:53,321 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-20 00:55:53,323 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.7010 sec 2023-03-20 00:55:54,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-20 00:55:54,742 INFO [Listener at localhost.localdomain/43361] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-03-20 00:55:54,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-03-20 00:55:54,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:55:54,757 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:55:54,759 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:55:54,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-20 00:55:54,764 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:54,767 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A, FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B, FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C, FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/recovered.edits] 2023-03-20 00:55:54,770 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/3e642b2ab3704e0b9687e1090201a4ca to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/3e642b2ab3704e0b9687e1090201a4ca 2023-03-20 00:55:54,772 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/94a3b82419a74025ba21398f9b90340a to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/94a3b82419a74025ba21398f9b90340a 2023-03-20 00:55:54,773 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/aa213358db7843f6b73b1f787cf41054 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/aa213358db7843f6b73b1f787cf41054 2023-03-20 00:55:54,775 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/adae2240e012406d9a76ca9e274bba1d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/adae2240e012406d9a76ca9e274bba1d 2023-03-20 00:55:54,776 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/e4441371ac374507b331e2f7ae179ede to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/A/e4441371ac374507b331e2f7ae179ede 2023-03-20 00:55:54,778 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/57805f0de05c40319cbc4dd42bad69cc to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/57805f0de05c40319cbc4dd42bad69cc 2023-03-20 00:55:54,779 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/a1b3cd032d57406db364eaab182cc974 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/a1b3cd032d57406db364eaab182cc974 2023-03-20 00:55:54,780 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/e31408ce2b274cc2af7ba44b0acd9ddb to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/e31408ce2b274cc2af7ba44b0acd9ddb 2023-03-20 00:55:54,782 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/f6a77fe47bb347c7bade58ad0701633d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/f6a77fe47bb347c7bade58ad0701633d 2023-03-20 00:55:54,783 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/f960f12aea5e4c8481a72242073dad37 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/B/f960f12aea5e4c8481a72242073dad37 2023-03-20 00:55:54,785 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/32fe4bc6d9674629b080d8d2af37cc8e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/32fe4bc6d9674629b080d8d2af37cc8e 2023-03-20 00:55:54,786 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/3cc69dd4dc5c41b38e0e42e50d55a5f7 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/3cc69dd4dc5c41b38e0e42e50d55a5f7 2023-03-20 00:55:54,787 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/8f7d019524bb4f7eb8dbc9e3b768b08c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/8f7d019524bb4f7eb8dbc9e3b768b08c 2023-03-20 00:55:54,788 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/a4fa96b75c8c459e89b3458f9b0ac74b to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/a4fa96b75c8c459e89b3458f9b0ac74b 2023-03-20 00:55:54,789 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/a770bc4695e040588b23d0e8c983981a to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/C/a770bc4695e040588b23d0e8c983981a 2023-03-20 00:55:54,792 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/recovered.edits/679.seqid to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98/recovered.edits/679.seqid 2023-03-20 00:55:54,793 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/1bfc994bd080bb1b8cbc4aef89c8cc98 2023-03-20 00:55:54,793 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-20 00:55:54,795 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:55:54,812 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-20 00:55:54,850 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-20 00:55:54,852 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:55:54,852 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-20 00:55:54,852 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679273754852"}]},"ts":"9223372036854775807"} 2023-03-20 00:55:54,854 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-20 00:55:54,854 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 1bfc994bd080bb1b8cbc4aef89c8cc98, NAME => 'TestAcidGuarantees,,1679273728917.1bfc994bd080bb1b8cbc4aef89c8cc98.', STARTKEY => '', ENDKEY => ''}] 2023-03-20 00:55:54,854 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-20 00:55:54,854 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679273754854"}]},"ts":"9223372036854775807"} 2023-03-20 00:55:54,856 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-20 00:55:54,857 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:55:54,858 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 109 msec 2023-03-20 00:55:54,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-20 00:55:54,861 INFO [Listener at localhost.localdomain/43361] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-03-20 00:55:54,881 INFO [Listener at localhost.localdomain/43361] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=234 (was 207) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5f8fccde-shared-pool-25 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-474632203_15 at /127.0.0.1:35676 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5f8fccde-shared-pool-23 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-474632203_15 at /127.0.0.1:35596 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5f8fccde-shared-pool-22 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5f8fccde-shared-pool-24 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase20:34553-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=545 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=334 (was 267) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 170), AvailableMemoryMB=4638 (was 5126) 2023-03-20 00:55:54,890 INFO [Listener at localhost.localdomain/43361] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=234, OpenFileDescriptor=545, MaxFileDescriptor=60000, SystemLoadAverage=334, ProcessCount=168, AvailableMemoryMB=4638 2023-03-20 00:55:54,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-20 00:55:54,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-20 00:55:54,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-20 00:55:54,895 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-20 00:55:54,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-03-20 00:55:54,896 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-20 00:55:54,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-20 00:55:54,899 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:55:54,899 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91 empty. 2023-03-20 00:55:54,900 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:55:54,900 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-20 00:55:54,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-20 00:55:55,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-20 00:55:55,313 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-20 00:55:55,316 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => f06a3c70774f5757abec4505bc3cdf91, NAME => 'TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp 2023-03-20 00:55:55,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-20 00:55:55,731 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 00:55:55,732 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing f06a3c70774f5757abec4505bc3cdf91, disabling compactions & flushes 2023-03-20 00:55:55,732 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:55,732 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:55,732 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. after waiting 0 ms 2023-03-20 00:55:55,732 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:55,732 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:55,732 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:55:55,738 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-20 00:55:55,740 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273755740"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679273755740"}]},"ts":"1679273755740"} 2023-03-20 00:55:55,742 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-20 00:55:55,744 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-20 00:55:55,744 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679273755744"}]},"ts":"1679273755744"} 2023-03-20 00:55:55,745 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-20 00:55:55,748 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f06a3c70774f5757abec4505bc3cdf91, ASSIGN}] 2023-03-20 00:55:55,750 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f06a3c70774f5757abec4505bc3cdf91, ASSIGN 2023-03-20 00:55:55,751 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=f06a3c70774f5757abec4505bc3cdf91, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34553,1679273725199; forceNewPlan=false, retain=false 2023-03-20 00:55:55,903 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=f06a3c70774f5757abec4505bc3cdf91, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:55,903 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273755903"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679273755903"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679273755903"}]},"ts":"1679273755903"} 2023-03-20 00:55:55,906 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199}] 2023-03-20 00:55:56,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-20 00:55:56,064 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:56,064 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f06a3c70774f5757abec4505bc3cdf91, NAME => 'TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.', STARTKEY => '', ENDKEY => ''} 2023-03-20 00:55:56,064 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:55:56,065 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 00:55:56,065 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:55:56,065 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:55:56,067 INFO [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:55:56,068 DEBUG [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A 2023-03-20 00:55:56,068 DEBUG [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A 2023-03-20 00:55:56,069 INFO [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 00:55:56,069 INFO [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f06a3c70774f5757abec4505bc3cdf91 columnFamilyName A 2023-03-20 00:55:56,069 INFO [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] regionserver.HStore(310): Store=f06a3c70774f5757abec4505bc3cdf91/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:55:56,069 INFO [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:55:56,070 DEBUG [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B 2023-03-20 00:55:56,070 DEBUG [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B 2023-03-20 00:55:56,071 INFO [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 00:55:56,071 INFO [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f06a3c70774f5757abec4505bc3cdf91 columnFamilyName B 2023-03-20 00:55:56,071 INFO [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] regionserver.HStore(310): Store=f06a3c70774f5757abec4505bc3cdf91/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:55:56,071 INFO [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:55:56,072 DEBUG [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C 2023-03-20 00:55:56,072 DEBUG [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C 2023-03-20 00:55:56,073 INFO [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 00:55:56,073 INFO [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f06a3c70774f5757abec4505bc3cdf91 columnFamilyName C 2023-03-20 00:55:56,073 INFO [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] regionserver.HStore(310): Store=f06a3c70774f5757abec4505bc3cdf91/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:55:56,073 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:56,074 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:55:56,075 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:55:56,077 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-20 00:55:56,078 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:55:56,080 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-20 00:55:56,080 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened f06a3c70774f5757abec4505bc3cdf91; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64083306, jitterRate=-0.04508432745933533}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-20 00:55:56,080 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:55:56,081 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91., pid=18, masterSystemTime=1679273756060 2023-03-20 00:55:56,082 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:56,082 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:56,083 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=f06a3c70774f5757abec4505bc3cdf91, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:56,083 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273756083"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679273756083"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679273756083"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679273756083"}]},"ts":"1679273756083"} 2023-03-20 00:55:56,086 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-03-20 00:55:56,086 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 in 178 msec 2023-03-20 00:55:56,089 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-03-20 00:55:56,089 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=f06a3c70774f5757abec4505bc3cdf91, ASSIGN in 338 msec 2023-03-20 00:55:56,090 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-20 00:55:56,090 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679273756090"}]},"ts":"1679273756090"} 2023-03-20 00:55:56,091 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-20 00:55:56,094 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-20 00:55:56,096 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2010 sec 2023-03-20 00:55:57,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-20 00:55:57,003 INFO [Listener at localhost.localdomain/43361] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-03-20 00:55:57,005 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x54ecd46c to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:55:57,011 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10a7f07a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:55:57,013 DEBUG [Listener at localhost.localdomain/43361] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:55:57,018 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:56162, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:55:57,020 DEBUG [Listener at localhost.localdomain/43361] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-20 00:55:57,023 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48184, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-20 00:55:57,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-20 00:55:57,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-20 00:55:57,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-20 00:55:57,455 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-20 00:55:57,456 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-20 00:55:57,456 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-20 00:55:57,461 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-20 00:55:57,476 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f06a3c70774f5757abec4505bc3cdf91, REOPEN/MOVE}] 2023-03-20 00:55:57,477 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f06a3c70774f5757abec4505bc3cdf91, REOPEN/MOVE 2023-03-20 00:55:57,478 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=f06a3c70774f5757abec4505bc3cdf91, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:57,478 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273757478"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679273757478"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679273757478"}]},"ts":"1679273757478"} 2023-03-20 00:55:57,479 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199}] 2023-03-20 00:55:57,632 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:55:57,633 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing f06a3c70774f5757abec4505bc3cdf91, disabling compactions & flushes 2023-03-20 00:55:57,633 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:57,633 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:57,633 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. after waiting 0 ms 2023-03-20 00:55:57,633 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:57,639 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-20 00:55:57,640 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:57,640 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:55:57,640 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: f06a3c70774f5757abec4505bc3cdf91 to self. 2023-03-20 00:55:57,642 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:55:57,643 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=f06a3c70774f5757abec4505bc3cdf91, regionState=CLOSED 2023-03-20 00:55:57,643 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273757643"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679273757643"}]},"ts":"1679273757643"} 2023-03-20 00:55:57,648 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-03-20 00:55:57,648 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 in 166 msec 2023-03-20 00:55:57,649 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=f06a3c70774f5757abec4505bc3cdf91, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,34553,1679273725199; forceNewPlan=false, retain=true 2023-03-20 00:55:57,799 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=f06a3c70774f5757abec4505bc3cdf91, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:57,800 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273757799"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679273757799"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679273757799"}]},"ts":"1679273757799"} 2023-03-20 00:55:57,802 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199}] 2023-03-20 00:55:57,965 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:57,965 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f06a3c70774f5757abec4505bc3cdf91, NAME => 'TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.', STARTKEY => '', ENDKEY => ''} 2023-03-20 00:55:57,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:55:57,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 00:55:57,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:55:57,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:55:57,970 INFO [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:55:57,971 DEBUG [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A 2023-03-20 00:55:57,971 DEBUG [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A 2023-03-20 00:55:57,972 INFO [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 00:55:57,975 INFO [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f06a3c70774f5757abec4505bc3cdf91 columnFamilyName A 2023-03-20 00:55:57,977 INFO [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] regionserver.HStore(310): Store=f06a3c70774f5757abec4505bc3cdf91/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:55:57,978 INFO [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:55:57,980 DEBUG [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B 2023-03-20 00:55:57,980 DEBUG [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B 2023-03-20 00:55:57,980 INFO [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 00:55:57,980 INFO [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f06a3c70774f5757abec4505bc3cdf91 columnFamilyName B 2023-03-20 00:55:57,981 INFO [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] regionserver.HStore(310): Store=f06a3c70774f5757abec4505bc3cdf91/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:55:57,981 INFO [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:55:57,982 DEBUG [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C 2023-03-20 00:55:57,982 DEBUG [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C 2023-03-20 00:55:57,982 INFO [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 00:55:57,983 INFO [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f06a3c70774f5757abec4505bc3cdf91 columnFamilyName C 2023-03-20 00:55:57,984 INFO [StoreOpener-f06a3c70774f5757abec4505bc3cdf91-1] regionserver.HStore(310): Store=f06a3c70774f5757abec4505bc3cdf91/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:55:57,984 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:57,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:55:57,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:55:57,989 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-20 00:55:57,990 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:55:57,991 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened f06a3c70774f5757abec4505bc3cdf91; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72483305, jitterRate=0.08008541166782379}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-20 00:55:57,991 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:55:57,992 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91., pid=23, masterSystemTime=1679273757955 2023-03-20 00:55:57,993 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:57,993 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:57,994 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=f06a3c70774f5757abec4505bc3cdf91, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:57,994 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273757994"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679273757994"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679273757994"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679273757994"}]},"ts":"1679273757994"} 2023-03-20 00:55:57,998 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-03-20 00:55:57,998 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 in 194 msec 2023-03-20 00:55:58,000 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-03-20 00:55:58,000 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=f06a3c70774f5757abec4505bc3cdf91, REOPEN/MOVE in 522 msec 2023-03-20 00:55:58,003 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-03-20 00:55:58,003 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 541 msec 2023-03-20 00:55:58,004 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 967 msec 2023-03-20 00:55:58,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-03-20 00:55:58,012 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x40eaa282 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:55:58,016 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b75e45b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:55:58,017 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x3b3311fc to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:55:58,026 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5649f410, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:55:58,026 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x595355a0 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:55:58,034 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7acc1387, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:55:58,035 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x7be739cc to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:55:58,044 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@61ad5d1c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:55:58,045 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x0a84dba6 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:55:58,053 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@486fba63, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:55:58,053 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x50feca84 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:55:58,061 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6577ce5a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:55:58,062 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x0bd79dff to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:55:58,070 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d5bdceb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:55:58,070 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x52902392 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:55:58,076 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@651f745a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:55:58,077 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x4e1da5ba to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:55:58,087 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d429358, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:55:58,093 DEBUG [hconnection-0x4b8d50eb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:55:58,093 DEBUG [hconnection-0x18bf9318-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:55:58,093 DEBUG [hconnection-0x56f5dfc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:55:58,093 DEBUG [hconnection-0x574aa420-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:55:58,093 DEBUG [hconnection-0x6066108c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:55:58,093 DEBUG [hconnection-0x615aaabb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:55:58,093 DEBUG [hconnection-0x15ffe6d6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:55:58,094 DEBUG [hconnection-0x3f2c2327-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:55:58,094 DEBUG [hconnection-0x1c4b3d04-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:55:58,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:58,097 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:56174, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:55:58,101 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:56176, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:55:58,101 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:56190, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:55:58,101 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:56182, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:55:58,101 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:56200, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:55:58,101 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:56218, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:55:58,102 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:56188, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:55:58,102 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:56226, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:55:58,102 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:56216, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:55:58,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:58,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:58,126 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:58,127 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:58,127 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:58,127 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:58,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:55:58,128 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f06a3c70774f5757abec4505bc3cdf91 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-20 00:55:58,129 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,129 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:58,129 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,130 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:58,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:58,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:58,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:58,130 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=A 2023-03-20 00:55:58,130 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:58,130 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=B 2023-03-20 00:55:58,130 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:58,130 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=C 2023-03-20 00:55:58,130 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:55:58,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:58,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:58,131 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:58,133 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,133 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:58,147 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:58,147 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:58,147 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:58,147 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:58,147 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:58,150 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:58,150 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:58,150 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:58,150 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:55:58,150 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:55:58,150 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:58,150 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:58,150 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:58,150 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679273758142, max timestamp=1679273758155Num uniques 859; 2023-03-20 00:55:58,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 00:55:58,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679273758142, max timestamp=1679273758156Num uniques 859; 2023-03-20 00:55:58,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 00:55:58,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679273758142, max timestamp=1679273758156Num uniques 859; 2023-03-20 00:55:58,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 00:55:58,170 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:58,170 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:58,171 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:58,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:58,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:58,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:58,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:58,174 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:58,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a572ad0[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:58,174 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:58,174 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,174 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,174 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:58,175 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,175 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:58,175 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:58,175 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:58,175 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,176 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:58,176 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:58,176 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,176 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,176 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,178 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:58,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273818171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,178 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:58,178 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:58,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273818172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56226 deadline: 1679273818173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:58,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:58,180 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:58,180 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:58,180 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:58,180 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:58,180 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:58,181 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,182 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:58,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,182 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:58,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:58,182 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,183 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:58,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273818178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:58,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:58,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:58,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:58,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273818178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,184 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:58,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:58,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,193 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,193 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:58,193 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:58,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:58,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:58,193 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:58,193 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:58,193 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:58,193 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:58,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:58,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:58,193 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:58,194 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:58,193 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,194 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,194 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:58,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:58,207 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320c708d36f37be4281be705cd7bde0177d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320c708d36f37be4281be705cd7bde0177d 2023-03-20 00:55:58,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:58,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273818280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:58,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273818280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,283 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:58,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56226 deadline: 1679273818280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,285 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:58,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273818284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,285 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:58,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273818285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,294 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:58,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:58,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:58,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:58,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:58,309 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:58,310 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:58,310 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:58,310 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:58,310 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,310 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:58,311 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:58,311 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:58,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:58,311 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,311 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:58,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:58,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:58,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:58,315 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:58,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:58,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:58,318 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:58,318 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:58,318 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:58,321 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:58,321 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:58,323 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:58,323 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:55:58,323 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:55:58,323 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:58,323 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:58,323 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:58,324 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,326 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:58,326 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:58,334 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,334 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:58,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:58,334 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,334 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:58,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:58,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:58,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:58,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:58,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79497fc1[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:58,336 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:58,336 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,337 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,337 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:58,337 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:58,337 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,337 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:58,337 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:58,338 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:58,339 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:58,339 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:58,339 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:58,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:58,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:58,341 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:58,341 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:58,341 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:58,341 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:58,341 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:58,351 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,351 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:58,351 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:58,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:58,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:58,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:58,352 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,353 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:58,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,359 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,359 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:58,359 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:58,359 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:58,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:58,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:58,359 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:58,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:58,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:58,359 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:58,359 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:58,359 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:58,359 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:58,359 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:58,359 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,360 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,460 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:58,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:58,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:58,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:58,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:58,470 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:58,470 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:58,470 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:58,470 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:58,471 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,471 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:58,472 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:58,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:58,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:58,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:58,472 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,473 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,473 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:58,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:58,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:58,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:58,473 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:58,473 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:58,473 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:58,474 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:58,474 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:58,478 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:58,478 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:58,478 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:58,478 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:55:58,478 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:55:58,478 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:58,478 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:58,478 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:58,478 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,478 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:58,478 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:58,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:58,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273818481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,486 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,487 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,487 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:58,487 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:58,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56226 deadline: 1679273818484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:58,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273818485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:58,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273818487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:58,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273818487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,490 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:58,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:58,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:58,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:58,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:58,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:58,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a0915fa[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:58,493 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:58,493 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,494 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,494 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:58,494 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:58,494 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:58,494 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,495 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:58,497 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:58,497 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,497 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:58,497 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:58,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:58,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:58,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:58,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:58,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:58,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:58,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:58,504 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:58,504 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:58,504 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:58,504 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:58,504 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:58,511 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,511 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,511 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:58,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:58,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:58,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:58,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:58,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:58,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:58,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:58,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,518 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,521 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,524 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,524 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:58,524 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:58,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:58,524 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:58,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:58,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:58,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:58,524 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:58,524 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:58,525 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,525 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,525 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,525 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,525 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:58,525 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:58,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:58,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:58,625 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:58,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:58,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:58,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:58,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:58,632 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:58,632 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:58,632 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:58,632 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/2cb67ad6701d47b48529ceb7dbac393f 2023-03-20 00:55:58,632 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:58,633 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,633 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:58,634 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:58,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:58,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:58,634 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:58,634 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,634 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:58,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:58,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:58,635 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:58,635 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:58,635 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:58,635 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:58,635 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:58,635 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:58,636 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:58,636 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:58,636 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:58,636 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:55:58,637 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:55:58,637 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:58,637 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:58,637 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:58,637 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,637 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:58,637 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:58,642 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,642 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:58,642 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,642 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:58,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:58,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:58,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:58,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61706e13[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:58,644 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:58,644 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,645 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,645 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:58,645 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,645 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:58,645 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:58,645 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:58,646 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:58,646 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,646 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:58,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:58,646 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:58,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:58,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:58,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:58,649 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:58,649 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:58,649 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:58,649 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:58,649 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:58,650 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,650 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:58,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,650 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:58,651 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,652 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,653 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:58,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:58,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:58,653 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:58,654 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:58,654 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:58,654 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,654 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:58,654 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:58,654 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,654 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,654 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,654 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:58,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:58,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:58,754 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:58,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:58,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:58,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:58,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:58,758 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:58,758 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:58,758 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:58,758 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:58,758 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,758 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:58,760 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,760 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:58,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:58,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:58,760 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,760 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:58,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:58,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:58,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:58,761 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:58,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:58,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:58,761 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:58,761 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:58,761 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:58,762 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:58,762 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:58,762 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:58,762 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:55:58,762 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:55:58,762 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:58,762 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:58,762 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:58,762 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,762 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:58,762 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:58,763 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,763 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:58,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:58,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:58,766 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,766 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:58,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:58,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:58,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:58,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32a2d947[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:58,768 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:58,768 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,770 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,770 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:58,770 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,772 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:58,772 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:58,772 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:58,775 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:58,775 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:58,775 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:58,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:58,775 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:58,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:58,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:58,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:58,779 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:58,779 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:58,779 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:58,779 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:58,779 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:58,781 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,781 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,781 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:58,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:58,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:58,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:58,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:58,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:58,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:58,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273818785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,786 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:58,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,788 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:58,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:58,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56226 deadline: 1679273818790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:58,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273818790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,794 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:58,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:58,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273818790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273818790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,800 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,800 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:58,800 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:58,800 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:58,800 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:58,800 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,800 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:58,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:58,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:58,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:58,800 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,801 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,800 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:58,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:58,801 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:58,801 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:58,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:58,875 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-20 00:55:58,901 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:58,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:58,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:58,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:58,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:58,918 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:58,918 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:58,918 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:58,918 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:58,919 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,919 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:58,919 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,919 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:58,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:58,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:58,920 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,920 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:58,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:58,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:58,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:58,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:58,921 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:58,921 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:58,921 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:58,921 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:58,921 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:58,921 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:58,921 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:58,921 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:55:58,921 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:55:58,921 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:58,922 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:58,922 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:58,922 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,922 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:58,922 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:58,923 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,923 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:58,923 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:58,923 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:58,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:58,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:58,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a71fa5[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:58,924 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:58,924 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,925 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,925 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,925 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:58,925 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:58,925 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:58,925 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:58,926 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:58,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:58,926 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:58,926 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:58,927 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:58,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:58,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:58,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:58,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:58,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:58,928 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:58,928 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:58,928 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:58,928 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:58,928 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:58,929 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,930 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:58,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:58,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:58,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,931 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:58,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:58,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:58,931 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:58,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,933 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:58,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:58,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,935 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,936 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:58,936 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:58,936 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:58,936 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:58,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:58,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:58,936 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:58,936 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,936 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:58,936 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:58,936 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:58,936 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:58,936 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:58,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:58,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:58,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:58,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:59,036 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:59,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:59,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:59,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:59,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:59,044 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:59,044 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:59,044 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:59,044 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:59,045 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,045 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:59,045 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:59,046 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:59,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:59,046 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,046 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:59,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:59,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:59,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:59,047 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:59,050 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:59,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:59,050 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:59,050 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:59,050 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:59,054 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:59,054 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:59,054 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:59,054 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:55:59,054 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:55:59,054 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:59,054 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:59,054 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:59,055 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,055 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:59,055 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:59,056 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,056 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:59,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:59,056 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,056 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:59,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:59,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:59,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:59,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:59,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:59,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22a36c8e[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:59,058 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:59,058 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,058 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,059 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:59,059 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:59,059 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:59,059 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,059 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:59,060 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:59,060 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:59,060 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:59,060 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:59,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:59,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:59,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:59,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:59,062 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:59,062 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:59,062 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:59,062 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:59,062 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:59,063 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:59,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:59,064 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,064 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/dae19c07a69e4966a217cfd1c4a37fd4 2023-03-20 00:55:59,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,064 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:59,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,066 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:59,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:59,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:59,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,069 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:59,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:59,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,073 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:59,073 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,073 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:59,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:59,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:59,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:59,073 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:59,073 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,073 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:59,073 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:59,073 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:59,073 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:59,073 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,073 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,073 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:59,173 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:59,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:59,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:59,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:59,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:59,178 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:59,178 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:59,178 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:59,178 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:59,178 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,178 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:59,180 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,180 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:59,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:59,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:59,180 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,180 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:59,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:59,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:59,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:59,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:59,181 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:59,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:59,182 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:59,182 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:59,182 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:59,182 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:59,182 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:59,182 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:59,183 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:55:59,183 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:55:59,183 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:59,183 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:59,183 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:59,183 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,183 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:59,183 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:59,184 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,184 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:59,185 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,185 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:59,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:59,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:59,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40c19e3f[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:59,186 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:59,186 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,189 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,189 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:59,189 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,190 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:59,190 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:59,190 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:59,191 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,191 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:59,191 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:59,191 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:59,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:59,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:59,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:59,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:59,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:59,193 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:59,193 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:59,193 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:59,193 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:59,193 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:59,196 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,196 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:59,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,196 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:59,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:59,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:59,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:59,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,198 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:59,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:59,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:59,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:59,200 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,202 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:59,203 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:59,203 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,203 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:59,203 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:59,203 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,203 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,203 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,203 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,203 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:59,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:59,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:59,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:59,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:59,203 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:59,203 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:59,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:59,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:59,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:59,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:59,290 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:59,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273819289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:59,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273819296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:59,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273819297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:59,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273819297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,297 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:55:59,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56226 deadline: 1679273819296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,303 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:59,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:59,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:59,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:59,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:59,307 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:59,307 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:59,307 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:59,307 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:59,308 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,308 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:59,308 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:59,308 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:59,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:59,309 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,309 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:59,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:59,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:59,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:59,310 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:59,310 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:59,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:59,310 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:59,310 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:59,310 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:59,313 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:59,313 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:59,313 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:59,313 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:55:59,313 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:55:59,313 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:59,313 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:59,313 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:59,313 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,313 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:59,313 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:59,317 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:59,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:59,317 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:59,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:59,318 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,318 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:59,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:59,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:59,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:59,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@648d1f81[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:59,319 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:59,319 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,320 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,320 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:59,320 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:59,320 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:59,320 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,320 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:59,321 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,321 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:59,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:59,321 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:59,321 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:59,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:59,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:59,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:59,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:59,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:59,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:59,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:59,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:59,324 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:59,324 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:59,324 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:59,324 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:59,324 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:59,325 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,325 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:59,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:59,325 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:59,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:59,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:59,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:59,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:59,326 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:59,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:59,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:59,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:59,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,329 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,336 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,336 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:59,336 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:59,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:59,336 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:59,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:59,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:59,336 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:59,336 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:59,336 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:59,336 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:59,336 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:59,336 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,336 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,336 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:59,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:59,436 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:59,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:59,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:59,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:59,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:59,439 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:59,440 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:59,440 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:59,440 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:59,440 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,440 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:59,441 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:59,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:59,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:59,441 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,441 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:59,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:59,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:59,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:59,442 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:59,442 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:59,442 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:59,442 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:59,442 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:59,442 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:59,442 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:59,442 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:59,442 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:59,442 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:55:59,442 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:55:59,442 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:59,442 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:55:59,442 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:55:59,442 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,442 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:55:59,442 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:55:59,443 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,443 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:59,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:59,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:59,443 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,443 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:55:59,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:59,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:59,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:59,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:55:59,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c6f7e82[Count = 0] remaining members to acquire global barrier 2023-03-20 00:55:59,451 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:55:59,451 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,452 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,452 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:55:59,452 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:55:59,452 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:55:59,452 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,453 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:55:59,453 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,453 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:55:59,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:59,453 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:59,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:59,453 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:55:59,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:59,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:59,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:59,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:59,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:59,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:55:59,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:55:59,455 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:55:59,455 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:55:59,455 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:55:59,455 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:55:59,455 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:55:59,456 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,456 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:55:59,456 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:59,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:55:59,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:59,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:59,457 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:55:59,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:59,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:55:59,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:59,458 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:55:59,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:55:59,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,469 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:59,469 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:59,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:59,469 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:55:59,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:55:59,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:55:59,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:55:59,469 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:55:59,469 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:59,470 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:59,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:55:59,470 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,470 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:55:59,470 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:55:59,470 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:55:59,470 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,470 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:55:59,501 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/f7ec1a873fb344e2ad1b78fd0b9cade4 2023-03-20 00:55:59,506 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/2cb67ad6701d47b48529ceb7dbac393f as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/2cb67ad6701d47b48529ceb7dbac393f 2023-03-20 00:55:59,510 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/2cb67ad6701d47b48529ceb7dbac393f, entries=150, sequenceid=15, filesize=25.1 K 2023-03-20 00:55:59,511 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/dae19c07a69e4966a217cfd1c4a37fd4 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/dae19c07a69e4966a217cfd1c4a37fd4 2023-03-20 00:55:59,526 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/dae19c07a69e4966a217cfd1c4a37fd4, entries=150, sequenceid=15, filesize=11.6 K 2023-03-20 00:55:59,527 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/f7ec1a873fb344e2ad1b78fd0b9cade4 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/f7ec1a873fb344e2ad1b78fd0b9cade4 2023-03-20 00:55:59,532 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/f7ec1a873fb344e2ad1b78fd0b9cade4, entries=150, sequenceid=15, filesize=11.6 K 2023-03-20 00:55:59,533 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for f06a3c70774f5757abec4505bc3cdf91 in 1405ms, sequenceid=15, compaction requested=false 2023-03-20 00:55:59,533 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:55:59,570 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:55:59,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:55:59,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:55:59,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:55:59,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:55:59,574 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:55:59,574 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:59,574 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:55:59,574 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:55:59,576 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,576 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:59,577 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,577 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:55:59,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:55:59,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:55:59,577 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:55:59,577 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:55:59,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:59,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:55:59,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:55:59,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:55:59,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:55:59,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:55:59,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:55:59,579 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:55:59,579 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:55:59,579 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:55:59,579 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:55:59,579 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:55:59,579 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:55:59,579 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:55:59,579 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f06a3c70774f5757abec4505bc3cdf91 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-20 00:55:59,580 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=A 2023-03-20 00:55:59,580 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:59,580 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=B 2023-03-20 00:55:59,580 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:59,580 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=C 2023-03-20 00:55:59,580 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:55:59,621 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303204f636a72bff44b6e93f52f2bce61ad2c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303204f636a72bff44b6e93f52f2bce61ad2c 2023-03-20 00:56:00,061 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=52, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/e72b0c7214c24ee1b9163a2773c4c3b0 2023-03-20 00:56:00,089 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/559f78dcdd6f4af493f2c79f01cd7997 2023-03-20 00:56:00,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:56:00,298 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:00,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:00,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56226 deadline: 1679273820306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,310 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:00,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:00,310 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:00,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273820310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273820310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273820307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:00,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273820314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:00,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273820411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:00,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273820411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:00,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273820411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,419 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:00,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273820418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,512 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/6387bb4c161f4446bb57ac2696d55f02 2023-03-20 00:56:00,516 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/e72b0c7214c24ee1b9163a2773c4c3b0 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/e72b0c7214c24ee1b9163a2773c4c3b0 2023-03-20 00:56:00,520 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/e72b0c7214c24ee1b9163a2773c4c3b0, entries=150, sequenceid=52, filesize=25.1 K 2023-03-20 00:56:00,521 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/559f78dcdd6f4af493f2c79f01cd7997 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/559f78dcdd6f4af493f2c79f01cd7997 2023-03-20 00:56:00,524 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/559f78dcdd6f4af493f2c79f01cd7997, entries=150, sequenceid=52, filesize=11.6 K 2023-03-20 00:56:00,525 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/6387bb4c161f4446bb57ac2696d55f02 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/6387bb4c161f4446bb57ac2696d55f02 2023-03-20 00:56:00,533 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/6387bb4c161f4446bb57ac2696d55f02, entries=150, sequenceid=52, filesize=11.6 K 2023-03-20 00:56:00,534 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=53.67 KB/54960 for f06a3c70774f5757abec4505bc3cdf91 in 955ms, sequenceid=52, compaction requested=false 2023-03-20 00:56:00,534 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:00,534 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:00,534 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:00,534 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:00,534 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,534 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:00,535 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:00,535 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,535 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:00,536 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:00,536 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:00,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:00,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6591be9d[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:00,537 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:00,537 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:00,537 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:00,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:00,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:00,537 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:00,538 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:00,538 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:00,538 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,538 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:00,540 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:00,540 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,540 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:00,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:00,540 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:00,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:00,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:00,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:00,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:00,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:00,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:00,544 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:00,544 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:00,544 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:00,544 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:00,544 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:00,553 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:00,553 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:00,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:00,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:00,553 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:00,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:00,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:00,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:00,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:00,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:00,554 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:00,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:00,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:00,557 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:00,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:00,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,559 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,559 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:00,559 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:00,559 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,559 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:00,559 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:00,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:00,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:00,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:00,559 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:00,559 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:00,560 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:00,560 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:00,560 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:00,560 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:00,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:00,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:56:00,625 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f06a3c70774f5757abec4505bc3cdf91 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-20 00:56:00,626 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=A 2023-03-20 00:56:00,626 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:00,626 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=B 2023-03-20 00:56:00,627 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:00,627 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=C 2023-03-20 00:56:00,627 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:00,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:00,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273820650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:00,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273820651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:00,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273820652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:00,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273820653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,660 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:00,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:00,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:00,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:00,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:00,665 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:00,665 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:00,665 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:00,665 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:00,665 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:00,665 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:00,666 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:00,666 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:00,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:00,666 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,667 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:00,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:00,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:00,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:00,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:00,667 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:00,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:00,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:00,682 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:00,682 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:00,682 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:00,690 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:00,690 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:00,690 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:00,690 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:00,690 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:00,690 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:00,691 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:00,691 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:00,691 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,691 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:00,691 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:00,692 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,692 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:00,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:00,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:00,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:00,693 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:00,693 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:00,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:00,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:00,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:00,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:00,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29edf476[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:00,695 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:00,695 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:00,696 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,696 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:00,696 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:00,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:00,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:00,696 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:00,696 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:00,696 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:00,698 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,698 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:00,698 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:00,698 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:00,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:00,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:00,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:00,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:00,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:00,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:00,701 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:00,701 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:00,702 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:00,702 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:00,702 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:00,702 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303208fb921e4084f4a089418863c83735500 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303208fb921e4084f4a089418863c83735500 2023-03-20 00:56:00,711 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:00,711 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:00,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:00,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:00,711 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:00,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:00,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:00,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:00,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:00,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:00,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:00,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:00,712 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:00,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:00,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:00,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,713 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:00,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,719 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:00,719 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:00,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:00,719 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:00,719 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:00,719 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,719 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:00,719 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:00,719 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:00,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:00,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:00,719 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:00,719 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:00,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:00,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:00,721 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=67, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/586b07b919e8477c99257d608daeea4f 2023-03-20 00:56:00,725 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:00,725 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:00,743 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/f71ebd84f6d4433985b19c884b2f2148 2023-03-20 00:56:00,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:00,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273820754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,765 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:00,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273820758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,765 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:00,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273820758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:00,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273820759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,794 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/e0d1ac94575f46518634ba724284fb27 2023-03-20 00:56:00,799 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/586b07b919e8477c99257d608daeea4f as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/586b07b919e8477c99257d608daeea4f 2023-03-20 00:56:00,805 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/586b07b919e8477c99257d608daeea4f, entries=200, sequenceid=67, filesize=31.9 K 2023-03-20 00:56:00,806 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/f71ebd84f6d4433985b19c884b2f2148 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/f71ebd84f6d4433985b19c884b2f2148 2023-03-20 00:56:00,810 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/f71ebd84f6d4433985b19c884b2f2148, entries=150, sequenceid=67, filesize=11.6 K 2023-03-20 00:56:00,811 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/e0d1ac94575f46518634ba724284fb27 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/e0d1ac94575f46518634ba724284fb27 2023-03-20 00:56:00,815 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/e0d1ac94575f46518634ba724284fb27, entries=150, sequenceid=67, filesize=11.6 K 2023-03-20 00:56:00,819 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for f06a3c70774f5757abec4505bc3cdf91 in 194ms, sequenceid=67, compaction requested=true 2023-03-20 00:56:00,819 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:00,819 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:00,819 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:56:00,819 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:00,819 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:56:00,820 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:00,821 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84178 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:56:00,821 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:56:00,821 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): f06a3c70774f5757abec4505bc3cdf91/A is initiating minor compaction (all files) 2023-03-20 00:56:00,821 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): f06a3c70774f5757abec4505bc3cdf91/B is initiating minor compaction (all files) 2023-03-20 00:56:00,821 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f06a3c70774f5757abec4505bc3cdf91/A in TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:00,821 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f06a3c70774f5757abec4505bc3cdf91/B in TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:00,821 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/dae19c07a69e4966a217cfd1c4a37fd4, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/559f78dcdd6f4af493f2c79f01cd7997, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/f71ebd84f6d4433985b19c884b2f2148] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp, totalSize=34.8 K 2023-03-20 00:56:00,821 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/2cb67ad6701d47b48529ceb7dbac393f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/e72b0c7214c24ee1b9163a2773c4c3b0, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/586b07b919e8477c99257d608daeea4f] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp, totalSize=82.2 K 2023-03-20 00:56:00,822 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting dae19c07a69e4966a217cfd1c4a37fd4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679273758118 2023-03-20 00:56:00,822 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 559f78dcdd6f4af493f2c79f01cd7997, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1679273758166 2023-03-20 00:56:00,823 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 2cb67ad6701d47b48529ceb7dbac393f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679273758118 2023-03-20 00:56:00,824 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting f71ebd84f6d4433985b19c884b2f2148, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1679273760621 2023-03-20 00:56:00,824 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting e72b0c7214c24ee1b9163a2773c4c3b0, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1679273758166 2023-03-20 00:56:00,825 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 586b07b919e8477c99257d608daeea4f, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1679273760307 2023-03-20 00:56:00,825 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:00,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:00,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:00,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:00,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:00,832 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:00,832 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:00,832 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:00,832 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:00,832 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:00,832 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:00,833 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:00,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:00,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:00,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:00,834 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:00,834 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:00,834 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:00,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:00,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:00,835 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:00,835 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:00,835 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:00,835 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:00,835 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:00,835 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:00,836 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:00,836 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:00,836 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:00,836 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:00,836 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): f06a3c70774f5757abec4505bc3cdf91#B#compaction#108 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:00,836 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f06a3c70774f5757abec4505bc3cdf91 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-20 00:56:00,836 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=A 2023-03-20 00:56:00,836 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:00,836 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=B 2023-03-20 00:56:00,836 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:00,836 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=C 2023-03-20 00:56:00,836 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:00,890 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): f06a3c70774f5757abec4505bc3cdf91#A#compaction#109 average throughput is 0.47 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:00,904 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/e5b3e937950b4b9e81722ba4600d924e as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/e5b3e937950b4b9e81722ba4600d924e 2023-03-20 00:56:00,914 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f06a3c70774f5757abec4505bc3cdf91/B of f06a3c70774f5757abec4505bc3cdf91 into e5b3e937950b4b9e81722ba4600d924e(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:00,914 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:00,914 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91., storeName=f06a3c70774f5757abec4505bc3cdf91/B, priority=13, startTime=1679273760819; duration=0sec 2023-03-20 00:56:00,914 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:00,914 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:56:00,916 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:56:00,916 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): f06a3c70774f5757abec4505bc3cdf91/C is initiating minor compaction (all files) 2023-03-20 00:56:00,916 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f06a3c70774f5757abec4505bc3cdf91/C in TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:00,916 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/f7ec1a873fb344e2ad1b78fd0b9cade4, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/6387bb4c161f4446bb57ac2696d55f02, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/e0d1ac94575f46518634ba724284fb27] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp, totalSize=34.8 K 2023-03-20 00:56:00,917 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting f7ec1a873fb344e2ad1b78fd0b9cade4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679273758118 2023-03-20 00:56:00,917 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 6387bb4c161f4446bb57ac2696d55f02, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1679273758166 2023-03-20 00:56:00,918 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting e0d1ac94575f46518634ba724284fb27, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1679273760621 2023-03-20 00:56:00,937 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): f06a3c70774f5757abec4505bc3cdf91#C#compaction#111 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:00,941 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032040333ac79d7349f88df816633dfde872 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032040333ac79d7349f88df816633dfde872 2023-03-20 00:56:00,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:56:00,973 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:00,990 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=84, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/246d67c0b06d420a9eb4f1e58997b850 2023-03-20 00:56:01,000 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/2986d963dbfa485c94c9afa6e983e299 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/2986d963dbfa485c94c9afa6e983e299 2023-03-20 00:56:01,005 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:01,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273820994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:01,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273820999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,008 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f06a3c70774f5757abec4505bc3cdf91/C of f06a3c70774f5757abec4505bc3cdf91 into 2986d963dbfa485c94c9afa6e983e299(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:01,008 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:01,008 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91., storeName=f06a3c70774f5757abec4505bc3cdf91/C, priority=13, startTime=1679273760820; duration=0sec 2023-03-20 00:56:01,008 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:01,008 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:01,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273821003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,008 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:01,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273821005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,015 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/8c2d5dca21be45cdaec09dff4412956c 2023-03-20 00:56:01,048 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/33bdd917dca6490f9ba0bd0cd33aaf46 2023-03-20 00:56:01,054 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/246d67c0b06d420a9eb4f1e58997b850 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/246d67c0b06d420a9eb4f1e58997b850 2023-03-20 00:56:01,059 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/246d67c0b06d420a9eb4f1e58997b850, entries=150, sequenceid=84, filesize=25.1 K 2023-03-20 00:56:01,060 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/8c2d5dca21be45cdaec09dff4412956c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/8c2d5dca21be45cdaec09dff4412956c 2023-03-20 00:56:01,066 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/8c2d5dca21be45cdaec09dff4412956c, entries=150, sequenceid=84, filesize=11.6 K 2023-03-20 00:56:01,067 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/33bdd917dca6490f9ba0bd0cd33aaf46 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/33bdd917dca6490f9ba0bd0cd33aaf46 2023-03-20 00:56:01,072 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/33bdd917dca6490f9ba0bd0cd33aaf46, entries=150, sequenceid=84, filesize=11.6 K 2023-03-20 00:56:01,073 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for f06a3c70774f5757abec4505bc3cdf91 in 236ms, sequenceid=84, compaction requested=false 2023-03-20 00:56:01,073 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:01,073 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:01,073 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:01,073 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:01,073 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,073 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:01,073 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:01,074 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,074 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:01,074 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,074 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:01,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:01,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:01,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@259087dd[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:01,076 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:01,076 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,077 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,077 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:01,077 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:01,077 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:01,078 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:01,078 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:01,078 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:01,079 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,079 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:01,079 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:01,079 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:01,079 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:01,079 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:01,079 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:01,079 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:01,080 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,080 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:01,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:01,080 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:01,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:01,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:01,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:01,081 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:01,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,083 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:01,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,086 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,086 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:01,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:01,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:01,086 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:01,087 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:01,086 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:01,087 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,087 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,087 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,087 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,086 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:01,087 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:01,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:01,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:01,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:01,087 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:01,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:01,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:01,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:56:01,110 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f06a3c70774f5757abec4505bc3cdf91 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-20 00:56:01,110 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=A 2023-03-20 00:56:01,110 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:01,110 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=B 2023-03-20 00:56:01,110 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:01,111 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=C 2023-03-20 00:56:01,111 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:01,180 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:01,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273821173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,180 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:01,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273821173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:01,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:01,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273821178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273821177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,187 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:01,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:01,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:01,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:01,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:01,193 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:01,193 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:01,193 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:01,193 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:01,194 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,194 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:01,194 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:01,194 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:01,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:01,195 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,195 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:01,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:01,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:01,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:01,196 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:01,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:01,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:01,201 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:01,201 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:01,201 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:01,202 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:01,202 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:01,202 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:01,202 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:01,203 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:01,203 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:01,203 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:01,203 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:01,203 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,203 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:01,203 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:01,215 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,215 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:01,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:01,215 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,215 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:01,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:01,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:01,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:01,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:01,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ac6683[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:01,217 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:01,217 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,219 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,219 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,219 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:01,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,219 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:01,219 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:01,219 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:01,220 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,220 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:01,220 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:01,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:01,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:01,220 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:01,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:01,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:01,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:01,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:01,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:01,222 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:01,222 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:01,222 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:01,222 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:01,222 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:01,224 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,224 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,224 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:01,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:01,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:01,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:01,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:01,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:01,225 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:01,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,227 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:01,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,236 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,236 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:01,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:01,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:01,236 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:01,236 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:01,236 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:01,236 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:01,236 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,236 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,236 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,236 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,236 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:01,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:01,236 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:01,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:01,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:01,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:01,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273821281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:01,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273821282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,285 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:01,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273821284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,288 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:01,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273821284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,337 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:01,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:01,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:01,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:01,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:01,341 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:01,341 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:01,341 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:01,341 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:01,341 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,341 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:01,342 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:01,342 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:01,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:01,342 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,342 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:01,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:01,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:01,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:01,343 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:01,358 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:01,358 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:01,358 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:01,358 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:01,358 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:01,374 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:01,374 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:01,374 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:01,374 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:01,375 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:01,375 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:01,375 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:01,375 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:01,375 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,375 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:01,375 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:01,376 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,376 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:01,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:01,377 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,377 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:01,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:01,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:01,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:01,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6aa1551d[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:01,379 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:01,379 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,380 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,380 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:01,380 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:01,380 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:01,380 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,380 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:01,381 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,381 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:01,381 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:01,381 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:01,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:01,384 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:01,384 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:01,384 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:01,384 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:01,384 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:01,385 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,385 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,385 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:01,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:01,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:01,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,386 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:01,389 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,394 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:01,394 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:01,394 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:01,394 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:01,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:01,394 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:01,395 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:01,395 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,395 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,395 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,395 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,395 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:01,395 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:01,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:01,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:01,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:01,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:01,419 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/f2404f38f4ec4aed8f911bdd5cc4b5b3 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/f2404f38f4ec4aed8f911bdd5cc4b5b3 2023-03-20 00:56:01,426 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f06a3c70774f5757abec4505bc3cdf91/A of f06a3c70774f5757abec4505bc3cdf91 into f2404f38f4ec4aed8f911bdd5cc4b5b3(size=25.2 K), total size for store is 50.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:01,426 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:01,426 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91., storeName=f06a3c70774f5757abec4505bc3cdf91/A, priority=13, startTime=1679273760819; duration=0sec 2023-03-20 00:56:01,426 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:01,485 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:01,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273821483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,487 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:01,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273821485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:01,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273821486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,495 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:01,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273821492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,495 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:01,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:01,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:01,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:01,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:01,522 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:01,522 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:01,522 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:01,522 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:01,523 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,523 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:01,524 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,524 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:01,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:01,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:01,524 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,524 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:01,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:01,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:01,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:01,525 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:01,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:01,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:01,542 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:01,542 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:01,542 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:01,547 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320f3864e092284432696023c5ff79d6d13 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320f3864e092284432696023c5ff79d6d13 2023-03-20 00:56:01,548 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:01,548 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:01,549 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:01,549 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:01,549 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:01,549 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:01,549 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:01,549 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:01,549 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,549 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:01,549 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:01,550 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:01,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:01,551 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,551 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:01,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:01,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:01,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:01,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:01,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c29527b[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:01,552 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:01,553 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,553 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,553 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:01,553 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:01,554 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:01,554 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,554 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:01,555 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:01,555 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:01,555 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,555 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:01,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:01,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:01,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:01,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:01,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:01,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:01,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:01,557 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:01,557 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:01,557 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:01,557 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:01,558 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:01,559 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,559 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:01,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,559 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:01,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:01,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:01,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:01,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:01,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,560 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:01,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,562 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:01,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,567 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,567 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:01,567 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:01,567 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,567 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,567 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,567 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,567 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:01,567 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:01,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:01,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:01,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:01,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:01,567 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:01,568 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:01,568 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:01,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:01,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:01,596 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=103, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/120a3bf4684e4ef4a290c70c40200a37 2023-03-20 00:56:01,629 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/8d01404fb75440938a95659791116f4f 2023-03-20 00:56:01,668 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:01,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:01,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:01,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:01,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:01,674 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:01,674 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:01,674 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:01,674 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:01,675 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,675 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:01,676 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:01,676 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:01,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:01,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:01,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:01,678 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:01,678 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,678 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:01,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:01,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:01,682 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:01,682 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:01,682 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:01,682 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:01,682 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:01,682 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:01,682 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:01,683 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:01,683 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:01,683 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:01,683 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:01,683 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,683 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:01,683 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:01,692 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,692 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:01,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:01,692 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,693 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:01,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:01,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:01,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:01,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:01,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@da6864c[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:01,694 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:01,694 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,695 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,695 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:01,695 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:01,695 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:01,696 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:01,696 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:01,696 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:01,697 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,697 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:01,697 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:01,697 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:01,697 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:01,697 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:01,697 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:01,697 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:01,699 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,699 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:01,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,699 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:01,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:01,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:01,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:01,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:01,700 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:01,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,702 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:01,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,712 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:01,712 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,712 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:01,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:01,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:01,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:01,712 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:01,712 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:01,712 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:01,712 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,712 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,712 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:01,713 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:01,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:01,712 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,713 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:01,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:01,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:01,789 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:01,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:01,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273821789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273821789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:01,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273821790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:01,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273821797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,813 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:01,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:01,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:01,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:01,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:01,826 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:01,826 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:01,826 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:01,826 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:01,828 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,828 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:01,829 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,829 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:01,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:01,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:01,829 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,829 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:01,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:01,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:01,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:01,831 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:01,844 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:01,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:01,844 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:01,844 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:01,844 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:01,844 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:01,844 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:01,844 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:01,844 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:01,844 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:01,844 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:01,845 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:01,845 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:01,845 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,845 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:01,845 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:01,852 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,852 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:01,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:01,853 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,853 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:01,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:01,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:01,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:01,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:01,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c5f7105[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:01,860 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:01,860 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,861 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,861 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:01,861 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:01,861 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:01,861 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,861 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:01,863 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:01,863 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:01,863 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:01,863 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:01,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:01,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:01,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:01,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:01,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:01,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:01,867 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:01,867 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:01,867 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:01,867 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:01,867 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:01,868 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,868 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:01,868 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:01,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:01,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:01,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:01,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:01,869 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:01,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,871 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:01,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:01,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,875 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,875 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:01,875 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:01,875 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,875 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,875 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:01,875 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:01,875 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:01,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:01,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:01,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:01,875 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:01,875 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:01,876 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:01,876 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:01,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:01,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:01,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:01,976 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:01,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:01,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:01,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:01,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:01,980 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:01,980 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:01,980 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:01,980 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:01,980 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,981 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:01,991 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:01,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:01,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:01,991 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,991 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,991 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:01,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:01,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:01,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:01,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:01,992 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:01,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:01,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:01,998 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:01,998 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:01,998 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:01,999 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:01,999 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:01,999 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:01,999 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:01,999 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:01,999 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:01,999 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:01,999 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:01,999 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:01,999 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:01,999 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:02,002 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,002 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:02,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:02,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:02,003 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:02,003 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:02,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:02,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:02,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:02,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:02,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:02,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c36d82e[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:02,004 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:02,004 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:02,005 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:02,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:02,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:02,005 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:02,005 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:02,005 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:02,005 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,005 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:02,006 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,006 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:02,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:02,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:02,006 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:02,006 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:02,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:02,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:02,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:02,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:02,008 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:02,008 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:02,008 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:02,008 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:02,008 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:02,009 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:02,009 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:02,009 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:02,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:02,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:02,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:02,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:02,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:02,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:02,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:02,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:02,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:02,015 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:02,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,016 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:02,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:02,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,019 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,019 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:02,019 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:02,019 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:02,019 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:02,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:02,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:02,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:02,019 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:02,020 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:02,019 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:02,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:02,020 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,020 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:02,020 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:02,020 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:02,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:02,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:02,070 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/00d0b06fb8f2479794cbfa227d36e210 2023-03-20 00:56:02,076 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/120a3bf4684e4ef4a290c70c40200a37 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/120a3bf4684e4ef4a290c70c40200a37 2023-03-20 00:56:02,081 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/120a3bf4684e4ef4a290c70c40200a37, entries=150, sequenceid=103, filesize=25.1 K 2023-03-20 00:56:02,082 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/8d01404fb75440938a95659791116f4f as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/8d01404fb75440938a95659791116f4f 2023-03-20 00:56:02,094 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/8d01404fb75440938a95659791116f4f, entries=150, sequenceid=103, filesize=11.6 K 2023-03-20 00:56:02,095 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/00d0b06fb8f2479794cbfa227d36e210 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/00d0b06fb8f2479794cbfa227d36e210 2023-03-20 00:56:02,099 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/00d0b06fb8f2479794cbfa227d36e210, entries=150, sequenceid=103, filesize=11.6 K 2023-03-20 00:56:02,100 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for f06a3c70774f5757abec4505bc3cdf91 in 990ms, sequenceid=103, compaction requested=true 2023-03-20 00:56:02,100 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:02,101 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:02,101 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:56:02,101 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:56:02,101 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:02,101 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:02,115 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77341 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:56:02,115 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:56:02,116 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): f06a3c70774f5757abec4505bc3cdf91/A is initiating minor compaction (all files) 2023-03-20 00:56:02,116 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): f06a3c70774f5757abec4505bc3cdf91/B is initiating minor compaction (all files) 2023-03-20 00:56:02,116 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f06a3c70774f5757abec4505bc3cdf91/A in TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:02,116 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f06a3c70774f5757abec4505bc3cdf91/B in TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:02,116 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/f2404f38f4ec4aed8f911bdd5cc4b5b3, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/246d67c0b06d420a9eb4f1e58997b850, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/120a3bf4684e4ef4a290c70c40200a37] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp, totalSize=75.5 K 2023-03-20 00:56:02,116 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/e5b3e937950b4b9e81722ba4600d924e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/8c2d5dca21be45cdaec09dff4412956c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/8d01404fb75440938a95659791116f4f] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp, totalSize=34.9 K 2023-03-20 00:56:02,117 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting f2404f38f4ec4aed8f911bdd5cc4b5b3, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1679273760621 2023-03-20 00:56:02,117 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 246d67c0b06d420a9eb4f1e58997b850, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1679273760637 2023-03-20 00:56:02,118 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting e5b3e937950b4b9e81722ba4600d924e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1679273760621 2023-03-20 00:56:02,118 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 120a3bf4684e4ef4a290c70c40200a37, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1679273761003 2023-03-20 00:56:02,118 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 8c2d5dca21be45cdaec09dff4412956c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1679273760637 2023-03-20 00:56:02,119 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 8d01404fb75440938a95659791116f4f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1679273761003 2023-03-20 00:56:02,120 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:02,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:02,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:02,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:02,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:02,126 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:02,126 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:02,126 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:02,126 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:02,127 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:02,127 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:02,128 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,128 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:02,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:02,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:02,128 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:02,128 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:02,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:02,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:02,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:02,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:02,131 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): f06a3c70774f5757abec4505bc3cdf91#B#compaction#118 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:02,133 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:02,133 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:02,133 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:02,133 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:02,133 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:02,144 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:02,144 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:02,150 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:02,150 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:02,150 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f06a3c70774f5757abec4505bc3cdf91 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-20 00:56:02,150 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=A 2023-03-20 00:56:02,150 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:02,150 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=B 2023-03-20 00:56:02,150 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:02,150 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=C 2023-03-20 00:56:02,151 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:02,157 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): f06a3c70774f5757abec4505bc3cdf91#A#compaction#117 average throughput is 0.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:02,201 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320c8e6e452dcdf4bb59d66b9bc0f327adf to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320c8e6e452dcdf4bb59d66b9bc0f327adf 2023-03-20 00:56:02,223 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=119, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/c5f782e1394b45238809df99261b49a4 2023-03-20 00:56:02,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:56:02,296 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:02,313 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:02,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273822311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:02,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273822313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:02,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273822313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:02,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56226 deadline: 1679273822313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:02,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273822314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,320 DEBUG [Thread-765] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4146 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91., hostname=jenkins-hbase20.apache.org,34553,1679273725199, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 00:56:02,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:02,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273822416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:02,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273822419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:02,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273822421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:02,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273822421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,613 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/2d80ed2d849d4b7991ef129825b9647d as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/2d80ed2d849d4b7991ef129825b9647d 2023-03-20 00:56:02,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:02,622 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f06a3c70774f5757abec4505bc3cdf91/B of f06a3c70774f5757abec4505bc3cdf91 into 2d80ed2d849d4b7991ef129825b9647d(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:02,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273822619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,623 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:02,624 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91., storeName=f06a3c70774f5757abec4505bc3cdf91/B, priority=13, startTime=1679273762101; duration=0sec 2023-03-20 00:56:02,624 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:02,624 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:56:02,625 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:56:02,626 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): f06a3c70774f5757abec4505bc3cdf91/C is initiating minor compaction (all files) 2023-03-20 00:56:02,626 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f06a3c70774f5757abec4505bc3cdf91/C in TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:02,626 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/2986d963dbfa485c94c9afa6e983e299, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/33bdd917dca6490f9ba0bd0cd33aaf46, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/00d0b06fb8f2479794cbfa227d36e210] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp, totalSize=34.9 K 2023-03-20 00:56:02,626 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 2986d963dbfa485c94c9afa6e983e299, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1679273760621 2023-03-20 00:56:02,631 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 33bdd917dca6490f9ba0bd0cd33aaf46, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1679273760637 2023-03-20 00:56:02,632 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 00d0b06fb8f2479794cbfa227d36e210, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1679273761003 2023-03-20 00:56:02,633 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:02,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273822624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,633 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:02,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273822624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:02,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273822627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,644 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/ddcf88331ff14d68a63d62ef4286644b 2023-03-20 00:56:02,660 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/71fa428390b14b0b95c14b139dc1a595 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/71fa428390b14b0b95c14b139dc1a595 2023-03-20 00:56:02,668 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f06a3c70774f5757abec4505bc3cdf91/A of f06a3c70774f5757abec4505bc3cdf91 into 71fa428390b14b0b95c14b139dc1a595(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:02,668 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:02,668 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91., storeName=f06a3c70774f5757abec4505bc3cdf91/A, priority=13, startTime=1679273762101; duration=0sec 2023-03-20 00:56:02,668 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:02,675 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): f06a3c70774f5757abec4505bc3cdf91#C#compaction#122 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:02,689 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/e9d4c3cd33af444f87b72e8859f6c10a 2023-03-20 00:56:02,702 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/c5f782e1394b45238809df99261b49a4 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/c5f782e1394b45238809df99261b49a4 2023-03-20 00:56:02,711 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/c5f782e1394b45238809df99261b49a4, entries=150, sequenceid=119, filesize=25.1 K 2023-03-20 00:56:02,713 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/ddcf88331ff14d68a63d62ef4286644b as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/ddcf88331ff14d68a63d62ef4286644b 2023-03-20 00:56:02,713 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/95de81ae673b4ee987c1af7925ef1d08 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/95de81ae673b4ee987c1af7925ef1d08 2023-03-20 00:56:02,718 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/ddcf88331ff14d68a63d62ef4286644b, entries=150, sequenceid=119, filesize=11.6 K 2023-03-20 00:56:02,719 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/e9d4c3cd33af444f87b72e8859f6c10a as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/e9d4c3cd33af444f87b72e8859f6c10a 2023-03-20 00:56:02,728 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f06a3c70774f5757abec4505bc3cdf91/C of f06a3c70774f5757abec4505bc3cdf91 into 95de81ae673b4ee987c1af7925ef1d08(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:02,728 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:02,728 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91., storeName=f06a3c70774f5757abec4505bc3cdf91/C, priority=13, startTime=1679273762101; duration=0sec 2023-03-20 00:56:02,728 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:02,731 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/e9d4c3cd33af444f87b72e8859f6c10a, entries=150, sequenceid=119, filesize=11.6 K 2023-03-20 00:56:02,731 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for f06a3c70774f5757abec4505bc3cdf91 in 581ms, sequenceid=119, compaction requested=false 2023-03-20 00:56:02,731 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:02,732 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:02,732 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:02,732 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:02,732 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,732 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:02,732 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:02,733 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,733 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:02,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:02,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:02,733 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:02,733 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:02,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:02,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:02,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:02,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:02,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:02,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a6af959[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:02,734 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:02,734 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:02,752 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:02,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:02,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:02,752 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:02,752 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:02,753 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:02,753 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,753 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:02,758 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,758 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:02,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,759 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:02,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:02,759 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:02,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:02,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:02,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:02,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:02,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:02,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:02,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:02,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:02,761 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:02,761 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:02,761 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:02,761 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:02,761 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:02,769 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:02,769 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:02,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:02,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:02,769 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:02,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:02,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:02,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:02,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:02,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:02,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:02,770 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:02,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:02,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:02,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:02,773 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:02,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:02,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,777 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:02,777 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:02,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:02,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:02,778 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,778 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:02,778 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:02,778 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:02,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:02,778 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:02,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:02,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:02,778 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:02,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:02,778 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:02,778 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:02,778 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:02,778 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:02,878 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:02,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:02,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:02,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:02,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:02,883 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:02,883 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:02,884 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:02,884 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:02,884 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:02,884 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:02,890 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:02,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:02,890 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:02,890 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,890 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:02,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:02,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:02,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:02,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:02,891 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:02,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:02,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:02,894 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:02,894 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:02,894 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:02,894 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:02,894 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:02,894 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:02,894 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:02,895 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f06a3c70774f5757abec4505bc3cdf91 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-20 00:56:02,895 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=A 2023-03-20 00:56:02,895 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:02,895 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=B 2023-03-20 00:56:02,895 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:02,895 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=C 2023-03-20 00:56:02,896 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:02,919 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320c908dfa62b0e42489d460d116431c6dd to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320c908dfa62b0e42489d460d116431c6dd 2023-03-20 00:56:02,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:56:02,925 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:02,927 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=139, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/8c669a1198e143f4a63435beb9880e6e 2023-03-20 00:56:02,944 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:02,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273822939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:02,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273822943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,948 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:02,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273822945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:02,948 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:02,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273822945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:03,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:03,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273823046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:03,050 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:03,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273823048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:03,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:03,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273823049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:03,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:03,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273823049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:03,253 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:03,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273823249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:03,253 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:03,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:03,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273823251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:03,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273823251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:03,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:03,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273823252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:03,342 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=139 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/8ed0bcc71f1f410bbaa8d3e11188501d 2023-03-20 00:56:03,557 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:03,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273823556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:03,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:03,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273823557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:03,560 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:03,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273823557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:03,560 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:03,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273823558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:03,778 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=139 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/8423f8f8a7a54f248a6d705ce555f65d 2023-03-20 00:56:03,788 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/8c669a1198e143f4a63435beb9880e6e as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/8c669a1198e143f4a63435beb9880e6e 2023-03-20 00:56:03,793 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/8c669a1198e143f4a63435beb9880e6e, entries=150, sequenceid=139, filesize=25.3 K 2023-03-20 00:56:03,794 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/8ed0bcc71f1f410bbaa8d3e11188501d as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/8ed0bcc71f1f410bbaa8d3e11188501d 2023-03-20 00:56:03,799 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/8ed0bcc71f1f410bbaa8d3e11188501d, entries=150, sequenceid=139, filesize=11.8 K 2023-03-20 00:56:03,800 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/8423f8f8a7a54f248a6d705ce555f65d as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/8423f8f8a7a54f248a6d705ce555f65d 2023-03-20 00:56:03,807 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/8423f8f8a7a54f248a6d705ce555f65d, entries=150, sequenceid=139, filesize=11.8 K 2023-03-20 00:56:03,808 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for f06a3c70774f5757abec4505bc3cdf91 in 913ms, sequenceid=139, compaction requested=true 2023-03-20 00:56:03,808 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:03,809 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:03,809 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:03,809 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:03,809 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:03,810 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:03,810 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:03,811 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:03,811 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:03,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:03,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:03,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:03,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:03,812 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:03,812 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:03,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:03,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:03,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:03,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:03,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:03,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@534786be[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:03,813 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:03,814 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:03,814 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:03,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:03,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:03,815 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:03,815 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:03,815 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:03,815 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:03,815 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:03,816 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:03,816 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:03,816 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:03,816 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:03,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:03,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:03,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:03,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:03,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:03,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:03,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:03,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:03,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:03,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:03,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:03,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:03,819 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:03,819 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:03,819 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:03,819 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:03,819 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:03,820 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:03,820 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:03,820 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:03,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:03,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:03,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:03,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:03,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:03,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:03,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:03,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:03,821 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:03,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:03,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:03,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:03,823 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:03,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:03,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:03,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:03,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:03,825 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:03,825 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:03,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:03,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:03,825 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:03,825 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:03,825 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:03,825 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:03,825 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:03,825 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:03,825 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:03,825 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:03,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:03,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:03,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:03,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:03,826 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:03,826 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:03,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:03,926 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:03,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:03,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:03,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:03,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:03,938 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:03,938 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:03,938 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:03,938 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:03,938 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:03,939 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:03,939 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:03,940 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:03,940 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:03,940 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:03,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:03,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:03,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:03,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:03,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:03,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:03,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:03,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:03,949 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:03,949 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:03,949 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:03,949 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:03,950 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:03,950 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:03,950 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:03,950 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:03,950 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f06a3c70774f5757abec4505bc3cdf91 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-20 00:56:03,950 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=A 2023-03-20 00:56:03,950 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:03,951 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=B 2023-03-20 00:56:03,951 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:03,951 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=C 2023-03-20 00:56:03,951 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:03,994 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032005924189bedd43bab1815fb6fd86c036 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032005924189bedd43bab1815fb6fd86c036 2023-03-20 00:56:04,012 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=155, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/5fb70bbd110a43e6b3347fd64598fb8c 2023-03-20 00:56:04,063 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/c2ade211bd694abebacaedeac504a3ec 2023-03-20 00:56:04,065 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:04,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:56:04,088 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:04,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273824083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,088 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:04,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273824083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:04,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273824086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:04,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273824086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:04,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273824189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,191 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:04,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273824189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:04,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273824190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:04,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273824190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,393 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:04,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273824393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,395 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:04,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273824393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,399 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:04,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273824398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,406 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:04,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273824400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,510 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/063f2a5c739047d2bb64b92e3b89aca2 2023-03-20 00:56:04,517 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/5fb70bbd110a43e6b3347fd64598fb8c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/5fb70bbd110a43e6b3347fd64598fb8c 2023-03-20 00:56:04,526 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/5fb70bbd110a43e6b3347fd64598fb8c, entries=150, sequenceid=155, filesize=25.3 K 2023-03-20 00:56:04,527 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/c2ade211bd694abebacaedeac504a3ec as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/c2ade211bd694abebacaedeac504a3ec 2023-03-20 00:56:04,532 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/c2ade211bd694abebacaedeac504a3ec, entries=150, sequenceid=155, filesize=11.8 K 2023-03-20 00:56:04,533 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/063f2a5c739047d2bb64b92e3b89aca2 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/063f2a5c739047d2bb64b92e3b89aca2 2023-03-20 00:56:04,544 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/063f2a5c739047d2bb64b92e3b89aca2, entries=150, sequenceid=155, filesize=11.8 K 2023-03-20 00:56:04,545 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for f06a3c70774f5757abec4505bc3cdf91 in 595ms, sequenceid=155, compaction requested=true 2023-03-20 00:56:04,545 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:04,545 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:04,546 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:04,546 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:04,546 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,546 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:04,546 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:04,547 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:04,547 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,547 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:04,547 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:04,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:04,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:04,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:04,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:04,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:04,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:04,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:04,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4487128a[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:04,549 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:04,549 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:04,549 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:04,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:04,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:04,549 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:04,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:04,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:04,550 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,550 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:04,551 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:04,551 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:04,551 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,551 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:04,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:04,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:04,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:04,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:04,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:04,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:04,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:04,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:04,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:04,553 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:04,553 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:04,553 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:04,553 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:04,553 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:04,555 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:04,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:04,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:04,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:04,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:04,555 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:04,556 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:04,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:04,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:04,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:04,556 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:04,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:04,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:04,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:04,557 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:04,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:04,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:04,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,559 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:04,559 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:04,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:04,559 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:04,559 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:04,559 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:04,559 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,559 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:04,559 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:04,559 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:04,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:04,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:04,559 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:04,559 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:04,559 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:04,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:04,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:04,659 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:04,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:04,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:04,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:04,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:04,666 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:04,666 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:04,666 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:04,666 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:04,667 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:04,667 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:04,667 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,667 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:04,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:04,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:04,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:04,668 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,668 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:04,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:04,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:04,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:04,669 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:04,669 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:04,669 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:04,669 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:04,669 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:04,669 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:04,670 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:04,670 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:04,670 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:04,670 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:04,670 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f06a3c70774f5757abec4505bc3cdf91 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-20 00:56:04,670 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=A 2023-03-20 00:56:04,671 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:04,671 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=B 2023-03-20 00:56:04,674 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:04,681 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=C 2023-03-20 00:56:04,681 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:04,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:56:04,700 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:04,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:04,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273824711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:04,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273824712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:04,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273824714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,717 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:04,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273824715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:04,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273824816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:04,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273824817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:04,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273824817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:04,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:04,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273824818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:05,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:05,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273825019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:05,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:05,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273825021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:05,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:05,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273825021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:05,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:05,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273825023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:05,103 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320ee4e18bfd58142f9abb61e703c2a010e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320ee4e18bfd58142f9abb61e703c2a010e 2023-03-20 00:56:05,323 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:05,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273825322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:05,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:05,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273825326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:05,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:05,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273825326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:05,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:05,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273825329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:05,508 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=171, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/87fd9a4aa80b4af695bd530873ac9d13 2023-03-20 00:56:05,835 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:05,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273825829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:05,835 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:05,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273825829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:05,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:05,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273825831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:05,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:05,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273825835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:05,920 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/8a09574f74f64107a70e91a1505512b6 2023-03-20 00:56:05,953 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/54a09d3700614249b1533c622787cfcb 2023-03-20 00:56:05,960 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/87fd9a4aa80b4af695bd530873ac9d13 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/87fd9a4aa80b4af695bd530873ac9d13 2023-03-20 00:56:05,966 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/87fd9a4aa80b4af695bd530873ac9d13, entries=150, sequenceid=171, filesize=25.3 K 2023-03-20 00:56:05,967 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/8a09574f74f64107a70e91a1505512b6 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/8a09574f74f64107a70e91a1505512b6 2023-03-20 00:56:05,973 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/8a09574f74f64107a70e91a1505512b6, entries=150, sequenceid=171, filesize=11.8 K 2023-03-20 00:56:05,976 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/54a09d3700614249b1533c622787cfcb as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/54a09d3700614249b1533c622787cfcb 2023-03-20 00:56:05,983 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/54a09d3700614249b1533c622787cfcb, entries=150, sequenceid=171, filesize=11.8 K 2023-03-20 00:56:05,984 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for f06a3c70774f5757abec4505bc3cdf91 in 1313ms, sequenceid=171, compaction requested=true 2023-03-20 00:56:05,984 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:05,984 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:05,984 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:05,984 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:05,984 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:05,986 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:05,986 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:05,987 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:05,987 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:05,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:05,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:05,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:05,987 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:05,988 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:05,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:05,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:05,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:05,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:05,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:05,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:05,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e4a559c[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:05,990 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:05,990 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:05,990 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:05,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:05,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:05,990 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:05,991 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:05,991 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:05,991 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:05,991 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:05,991 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:05,991 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:05,992 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:05,992 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:05,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:05,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:05,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:05,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:05,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:05,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:05,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:05,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:05,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:05,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:05,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:05,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:05,994 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:05,995 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:05,995 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:05,995 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:05,995 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:05,996 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:05,996 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:05,996 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:05,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:05,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:05,998 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:05,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:05,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:05,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:05,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:05,999 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:05,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:05,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:06,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:06,001 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:06,001 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:06,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:06,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:06,001 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:06,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:06,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:06,001 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:06,001 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:06,001 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:06,001 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:06,001 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:06,002 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:06,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:06,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:06,002 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:06,002 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:06,002 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:06,102 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:06,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:06,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:06,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:06,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:06,107 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:06,107 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:06,107 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:06,107 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:06,118 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:06,118 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:06,119 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:06,119 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:06,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:06,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:06,119 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:06,119 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:06,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:06,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:06,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:06,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:06,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:06,133 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:06,133 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:06,133 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:06,133 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:06,133 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:06,134 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:06,134 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:06,134 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:06,134 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:06,134 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f06a3c70774f5757abec4505bc3cdf91 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-20 00:56:06,134 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=A 2023-03-20 00:56:06,135 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:06,135 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=B 2023-03-20 00:56:06,135 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:06,135 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=C 2023-03-20 00:56:06,135 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:06,325 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:06,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:56:06,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:06,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56226 deadline: 1679273826373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:06,479 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:06,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56226 deadline: 1679273826477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:06,569 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303207e26244bbaa34ded99c9ca8f47acac71 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303207e26244bbaa34ded99c9ca8f47acac71 2023-03-20 00:56:06,685 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:06,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56226 deadline: 1679273826681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:06,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:06,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273826841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:06,848 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:06,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273826843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:06,848 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:06,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273826846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:06,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:06,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273826846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:06,976 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=187, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/7cd8c20e3c4d4da2b4057fab3eea61d3 2023-03-20 00:56:06,991 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:06,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56226 deadline: 1679273826988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:07,388 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/940e7d64c004464882f4ab983c21f277 2023-03-20 00:56:07,497 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:07,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56226 deadline: 1679273827494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:07,802 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/c1d17a093eeb440e8a99f00af145f8bc 2023-03-20 00:56:07,807 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/7cd8c20e3c4d4da2b4057fab3eea61d3 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/7cd8c20e3c4d4da2b4057fab3eea61d3 2023-03-20 00:56:07,818 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/7cd8c20e3c4d4da2b4057fab3eea61d3, entries=150, sequenceid=187, filesize=25.3 K 2023-03-20 00:56:07,820 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/940e7d64c004464882f4ab983c21f277 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/940e7d64c004464882f4ab983c21f277 2023-03-20 00:56:07,833 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/940e7d64c004464882f4ab983c21f277, entries=150, sequenceid=187, filesize=11.8 K 2023-03-20 00:56:07,836 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/c1d17a093eeb440e8a99f00af145f8bc as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/c1d17a093eeb440e8a99f00af145f8bc 2023-03-20 00:56:07,843 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/c1d17a093eeb440e8a99f00af145f8bc, entries=150, sequenceid=187, filesize=11.8 K 2023-03-20 00:56:07,845 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for f06a3c70774f5757abec4505bc3cdf91 in 1711ms, sequenceid=187, compaction requested=true 2023-03-20 00:56:07,845 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:07,845 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:07,845 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:07,845 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:07,845 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:07,848 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:07,848 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:07,851 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:07,851 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:07,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:07,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:07,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:07,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:07,852 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:07,852 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:07,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:07,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:07,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:07,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:07,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:07,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43d7db3d[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:07,853 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:07,853 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:07,854 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:07,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:07,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:07,854 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:07,854 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:07,854 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:07,854 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:07,854 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:07,855 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:07,855 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:07,855 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:07,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:07,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:07,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:07,855 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:07,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:07,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:07,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:07,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:07,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:07,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:07,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:07,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:07,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:07,858 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:07,858 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:07,858 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:07,858 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:07,858 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:07,859 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:07,859 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:07,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:07,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:07,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:07,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:07,859 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:07,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:07,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:07,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:07,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:07,860 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:07,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:07,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:07,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:07,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:07,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:07,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:07,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:07,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:07,866 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:07,879 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:07,879 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:07,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:07,879 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:07,879 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:07,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:07,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:07,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:07,880 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:07,880 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:07,879 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:07,880 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:07,880 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:07,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:07,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:07,880 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:07,880 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:07,880 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:07,980 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:07,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:07,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:07,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:07,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:07,989 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:07,989 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:07,989 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:07,989 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:07,989 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:07,989 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:07,990 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:07,990 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:07,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:07,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:07,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:07,991 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:07,991 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:07,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:07,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:07,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:07,991 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:07,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:07,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:07,998 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:07,998 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:07,998 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:07,999 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:07,999 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:07,999 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:07,999 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:07,999 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f06a3c70774f5757abec4505bc3cdf91 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-20 00:56:07,999 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=A 2023-03-20 00:56:07,999 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:07,999 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=B 2023-03-20 00:56:08,000 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:08,000 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=C 2023-03-20 00:56:08,000 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:08,027 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303206c87dae1870b4fe9ad15c420e4443449 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303206c87dae1870b4fe9ad15c420e4443449 2023-03-20 00:56:08,436 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=203, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/726552578d4546bb83d305563cb29167 2023-03-20 00:56:08,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:56:08,507 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:08,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:08,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56226 deadline: 1679273828553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:08,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:08,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56226 deadline: 1679273828654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:08,847 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/377a137bb2554784a942f581826b358b 2023-03-20 00:56:08,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:08,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273828852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:08,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:08,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273828852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:08,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:08,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273828854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:08,860 DEBUG [Thread-771] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4141 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91., hostname=jenkins-hbase20.apache.org,34553,1679273725199, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 00:56:08,860 DEBUG [Thread-769] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91., hostname=jenkins-hbase20.apache.org,34553,1679273725199, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 00:56:08,860 DEBUG [Thread-763] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91., hostname=jenkins-hbase20.apache.org,34553,1679273725199, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 00:56:08,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:08,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56226 deadline: 1679273828863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:08,866 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/24741bbcd5e2464a8f9705c08137ac65 2023-03-20 00:56:08,871 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/726552578d4546bb83d305563cb29167 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/726552578d4546bb83d305563cb29167 2023-03-20 00:56:08,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:08,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273828871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:08,874 DEBUG [Thread-767] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4159 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91., hostname=jenkins-hbase20.apache.org,34553,1679273725199, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 00:56:08,878 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/726552578d4546bb83d305563cb29167, entries=150, sequenceid=203, filesize=25.3 K 2023-03-20 00:56:08,879 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/377a137bb2554784a942f581826b358b as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/377a137bb2554784a942f581826b358b 2023-03-20 00:56:08,888 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/377a137bb2554784a942f581826b358b, entries=150, sequenceid=203, filesize=11.8 K 2023-03-20 00:56:08,889 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/24741bbcd5e2464a8f9705c08137ac65 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/24741bbcd5e2464a8f9705c08137ac65 2023-03-20 00:56:08,896 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/24741bbcd5e2464a8f9705c08137ac65, entries=150, sequenceid=203, filesize=11.8 K 2023-03-20 00:56:08,897 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for f06a3c70774f5757abec4505bc3cdf91 in 898ms, sequenceid=203, compaction requested=true 2023-03-20 00:56:08,897 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:08,897 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:08,897 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:08,897 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:08,897 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:08,898 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:08,898 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:08,899 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:08,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:08,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:08,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:08,899 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:08,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:08,899 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:08,899 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:08,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:08,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:08,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:08,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:08,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:08,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c4ac78f[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:08,900 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:08,900 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:08,901 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:08,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:08,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:08,901 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:08,901 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:08,901 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:08,901 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:08,901 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:08,902 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:08,902 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:08,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:08,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:08,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:08,902 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:08,902 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:08,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:08,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:08,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:08,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:08,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:08,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:08,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:08,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:08,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:08,904 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:08,904 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:08,904 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:08,904 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:08,904 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:08,905 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:08,905 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:08,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:08,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:08,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:08,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:08,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:08,905 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:08,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:08,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:08,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:08,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:08,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:08,906 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:08,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:08,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:08,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:08,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:08,910 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:08,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:08,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:08,917 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:08,917 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:08,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:08,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:08,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:08,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:08,917 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:08,917 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:08,917 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:08,917 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:08,917 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:08,917 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:08,917 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:08,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:08,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:08,917 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:08,918 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:08,918 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:09,017 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:09,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:09,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:09,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:09,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:09,020 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:09,020 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:09,020 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:09,020 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:09,021 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:09,021 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:09,021 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:09,021 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:09,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:09,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:09,022 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:09,022 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:09,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:09,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:09,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:09,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:09,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:09,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:09,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:09,023 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:09,023 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:09,023 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:09,023 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:09,023 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:09,023 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:09,023 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:09,023 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f06a3c70774f5757abec4505bc3cdf91 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-20 00:56:09,023 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=A 2023-03-20 00:56:09,023 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:09,023 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=B 2023-03-20 00:56:09,024 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:09,024 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=C 2023-03-20 00:56:09,024 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:09,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:56:09,172 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:09,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:09,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56226 deadline: 1679273829228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:09,331 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:09,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56226 deadline: 1679273829330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:09,436 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303209384435f1c0a4aeda086bcc4949fd05d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303209384435f1c0a4aeda086bcc4949fd05d 2023-03-20 00:56:09,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:09,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56226 deadline: 1679273829532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:09,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:09,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56226 deadline: 1679273829835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:09,841 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=219, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/77cb88c105f44de18ff0e9e3bfad35b8 2023-03-20 00:56:10,251 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/9d39977c9f264cc08028f3c8e8e1e30c 2023-03-20 00:56:10,262 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/382486d354594cb5b71fd0cb7b97da9a 2023-03-20 00:56:10,266 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/77cb88c105f44de18ff0e9e3bfad35b8 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/77cb88c105f44de18ff0e9e3bfad35b8 2023-03-20 00:56:10,270 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/77cb88c105f44de18ff0e9e3bfad35b8, entries=150, sequenceid=219, filesize=25.3 K 2023-03-20 00:56:10,271 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/9d39977c9f264cc08028f3c8e8e1e30c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/9d39977c9f264cc08028f3c8e8e1e30c 2023-03-20 00:56:10,275 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/9d39977c9f264cc08028f3c8e8e1e30c, entries=150, sequenceid=219, filesize=11.8 K 2023-03-20 00:56:10,275 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/382486d354594cb5b71fd0cb7b97da9a as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/382486d354594cb5b71fd0cb7b97da9a 2023-03-20 00:56:10,280 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/382486d354594cb5b71fd0cb7b97da9a, entries=150, sequenceid=219, filesize=11.8 K 2023-03-20 00:56:10,280 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for f06a3c70774f5757abec4505bc3cdf91 in 1257ms, sequenceid=219, compaction requested=true 2023-03-20 00:56:10,280 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:10,280 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:10,281 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:10,281 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:10,281 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,282 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:10,282 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:10,283 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,283 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:10,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:10,283 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,283 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:10,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:10,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:10,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:10,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:10,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50a593ce[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:10,284 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:10,284 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,285 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,285 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:10,285 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:10,285 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:10,285 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,285 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:10,286 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:10,286 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:10,286 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:10,286 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:10,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:10,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:10,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:10,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:10,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:10,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:10,289 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:10,289 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:10,289 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:10,289 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:10,289 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:10,290 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,290 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:10,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:10,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:10,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,291 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:10,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:10,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:10,292 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:10,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,292 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:10,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,294 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:10,294 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:10,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:10,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:10,294 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,294 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:10,294 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:10,294 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,294 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,294 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,294 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,304 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:10,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:10,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:10,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:10,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:10,304 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:10,304 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:10,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:56:10,341 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f06a3c70774f5757abec4505bc3cdf91 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-20 00:56:10,341 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=A 2023-03-20 00:56:10,341 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:10,341 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=B 2023-03-20 00:56:10,341 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:10,341 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=C 2023-03-20 00:56:10,341 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:10,404 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:10,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56226 deadline: 1679273830404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,404 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:10,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:10,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:10,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:10,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:10,407 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:10,407 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:10,408 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:10,408 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:10,408 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,408 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:10,409 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:10,409 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:10,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:10,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:10,409 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,409 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:10,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:10,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:10,410 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:10,410 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:10,410 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:10,410 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:10,410 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:10,410 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:10,414 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:10,414 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:10,414 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:10,414 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:10,414 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:10,414 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:10,414 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:10,414 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:10,414 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,414 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:10,414 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:10,419 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,419 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:10,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:10,419 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,419 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:10,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:10,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:10,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:10,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:10,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cefdaf3[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:10,421 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:10,421 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,421 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,421 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:10,421 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:10,421 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:10,424 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:10,424 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:10,424 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:10,424 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,424 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,424 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:10,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:10,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:10,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:10,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:10,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:10,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:10,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:10,427 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:10,427 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:10,427 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:10,427 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:10,427 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:10,436 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,436 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,436 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:10,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:10,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:10,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:10,437 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:10,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:10,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:10,438 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:10,439 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,439 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:10,439 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:10,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:10,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:10,439 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:10,439 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,439 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,439 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,440 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,439 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:10,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:10,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:10,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:10,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:10,440 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:10,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:10,440 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:10,440 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:10,505 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:10,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56226 deadline: 1679273830505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,540 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:10,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:10,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:10,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:10,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:10,550 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:10,550 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:10,550 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:10,550 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:10,550 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,550 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:10,554 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:10,554 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:10,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:10,555 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,555 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:10,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:10,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:10,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:10,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:10,558 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:10,558 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:10,558 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:10,558 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:10,558 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:10,563 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:10,564 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:10,564 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:10,564 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:10,564 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:10,564 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:10,564 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:10,564 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:10,564 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,564 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:10,564 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:10,575 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,575 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:10,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:10,575 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,575 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:10,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:10,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:10,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:10,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:10,577 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:10,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19ce860[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:10,577 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,577 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,577 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:10,577 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:10,578 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:10,578 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:10,578 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:10,578 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:10,578 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,579 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:10,579 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:10,579 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:10,579 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:10,579 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:10,579 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:10,579 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:10,581 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,581 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:10,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,581 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:10,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:10,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,582 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:10,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:10,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:10,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:10,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,583 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:10,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,584 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,585 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:10,585 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:10,585 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:10,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:10,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:10,585 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:10,585 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:10,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:10,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:10,585 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:10,585 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,585 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,585 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:10,585 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:10,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:10,585 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:10,685 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:10,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:10,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:10,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:10,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:10,688 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:10,688 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:10,688 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:10,688 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:10,689 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,689 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:10,689 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:10,689 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:10,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:10,690 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,690 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:10,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:10,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:10,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:10,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:10,691 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:10,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:10,691 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:10,691 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:10,691 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:10,691 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:10,691 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:10,691 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:10,691 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:10,691 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:10,691 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:10,691 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:10,691 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:10,691 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,691 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:10,691 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:10,692 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,692 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:10,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:10,692 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,692 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:10,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:10,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:10,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:10,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:10,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b902163[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:10,695 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:10,695 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,696 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,696 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:10,697 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,697 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:10,697 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:10,697 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:10,698 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:10,698 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:10,698 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:10,698 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:10,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:10,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:10,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:10,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:10,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:10,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:10,700 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:10,700 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:10,700 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:10,700 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:10,700 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:10,700 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,700 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:10,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:10,701 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:10,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,701 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:10,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:10,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:10,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:10,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,702 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:10,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,703 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,703 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:10,703 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:10,703 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:10,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:10,703 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:10,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:10,704 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:10,704 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:10,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:10,704 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:10,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:10,704 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:10,704 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,704 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:10,704 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:10,713 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:10,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56226 deadline: 1679273830706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,760 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032027967a04bd1b4f21aa4973def16a3a80 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032027967a04bd1b4f21aa4973def16a3a80 2023-03-20 00:56:10,804 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:10,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:10,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:10,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:10,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:10,808 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:10,808 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:10,808 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:10,808 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:10,808 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,808 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:10,809 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,809 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:10,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:10,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:10,809 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,809 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:10,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:10,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:10,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:10,810 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:10,810 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:10,810 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:10,810 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:10,810 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:10,810 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:10,810 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:10,810 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:10,810 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:10,810 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:10,810 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:10,810 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:10,810 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:10,810 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:10,810 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,810 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:10,810 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:10,811 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,811 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:10,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:10,811 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,811 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:10,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:10,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:10,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:10,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:10,813 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:10,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@757f63c[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:10,813 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,813 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,813 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:10,813 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:10,813 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:10,813 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,813 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:10,814 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,814 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:10,814 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:10,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:10,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:10,814 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:10,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:10,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:10,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:10,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:10,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:10,816 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:10,816 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:10,816 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:10,816 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:10,816 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:10,817 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,817 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:10,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:10,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,817 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:10,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,817 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:10,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:10,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:10,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:10,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,818 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:10,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,819 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,819 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:10,819 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:10,819 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:10,819 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:10,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:10,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:10,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:10,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:10,819 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:10,819 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,819 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,819 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:10,820 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:10,820 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:10,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:10,820 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:10,920 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:10,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:10,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:10,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:10,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:10,923 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:10,923 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:10,923 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:10,923 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:10,923 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,923 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:10,924 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:10,924 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:10,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:10,924 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,924 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:10,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:10,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:10,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:10,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:10,925 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:10,925 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:10,925 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:10,925 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:10,926 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:10,926 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:10,926 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:10,927 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:10,927 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:10,927 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:10,927 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:10,927 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:10,927 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:10,927 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,927 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:10,927 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:10,933 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,933 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:10,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:10,934 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,934 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:10,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:10,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:10,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:10,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:10,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56d9c67[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:10,935 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:10,935 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,935 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,936 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:10,936 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:10,936 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:10,936 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,936 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:10,936 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:10,936 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,936 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:10,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:10,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:10,936 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:10,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:10,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:10,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:10,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:10,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:10,941 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:10,941 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:10,941 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:10,941 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:10,941 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:10,944 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,944 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,944 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:10,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:10,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:10,944 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:10,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:10,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:10,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:10,947 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:10,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:10,950 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,950 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:10,950 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:10,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:10,950 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:10,950 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:10,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:10,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:10,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:10,950 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:10,950 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:10,950 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,950 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:10,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:10,950 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:10,951 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:10,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:10,951 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:10,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:11,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:11,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56226 deadline: 1679273831016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,051 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:11,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:11,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:11,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:11,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:11,054 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:11,054 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:11,054 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:11,054 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:11,054 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,054 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,055 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:11,055 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:11,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:11,055 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,055 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:11,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:11,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:11,057 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:11,057 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:11,057 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:11,057 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:11,057 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:11,057 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:11,057 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:11,057 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:11,057 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:11,057 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:11,057 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:11,057 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:11,057 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:11,057 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,057 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:11,057 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:11,059 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,059 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:11,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:11,059 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:11,059 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:11,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:11,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:11,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:11,060 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:11,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6354bb42[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:11,060 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,061 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,061 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:11,061 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:11,061 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:11,061 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,061 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:11,062 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,062 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:11,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:11,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:11,062 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:11,062 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:11,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:11,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:11,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:11,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:11,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:11,064 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:11,064 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:11,064 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:11,064 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:11,064 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:11,065 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:11,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:11,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:11,066 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:11,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:11,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,067 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,068 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,068 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:11,070 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:11,070 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,070 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:11,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,070 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:11,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:11,070 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,070 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,070 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,070 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,070 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:11,070 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:11,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:11,070 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:11,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:11,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:11,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:11,165 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=236, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/3b5295f32bbc46099a6c416648abeeb6 2023-03-20 00:56:11,170 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:11,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:11,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:11,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:11,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:11,173 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:11,173 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:11,173 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:11,173 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:11,174 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,174 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,174 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:11,175 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:11,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:11,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,175 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,175 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:11,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:11,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:11,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:11,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:11,176 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:11,176 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:11,176 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:11,181 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:11,181 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:11,181 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:11,181 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:11,181 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:11,181 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:11,181 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:11,182 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:11,182 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,184 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:11,184 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:11,192 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,192 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:11,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:11,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:11,192 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,192 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:11,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:11,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:11,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:11,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19351572[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:11,193 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:11,193 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,194 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,194 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:11,194 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,194 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:11,194 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:11,194 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:11,195 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,195 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:11,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:11,195 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:11,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:11,195 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:11,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:11,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:11,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:11,198 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:11,199 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:11,199 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:11,199 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:11,199 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:11,200 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,200 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:11,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,200 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:11,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:11,200 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:11,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,202 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:11,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,206 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,206 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,206 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,206 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,206 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:11,206 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:11,206 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:11,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:11,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:11,206 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,206 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,206 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:11,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:11,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:11,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:11,207 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:11,207 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:11,307 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:11,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:11,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:11,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:11,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:11,310 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:11,310 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:11,310 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:11,310 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:11,311 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,311 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,311 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,311 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:11,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:11,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:11,312 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,312 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:11,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:11,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:11,314 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:11,314 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:11,314 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:11,315 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:11,315 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:11,318 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:11,318 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:11,318 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:11,318 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:11,318 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:11,318 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:11,318 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:11,318 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:11,318 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,318 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:11,318 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:11,319 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,319 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:11,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:11,320 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,320 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:11,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:11,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:11,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:11,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:11,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21ebf67e[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:11,321 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:11,321 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,321 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,321 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:11,321 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:11,321 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,321 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:11,321 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:11,322 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,322 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:11,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:11,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:11,322 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:11,322 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:11,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:11,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:11,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:11,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:11,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:11,324 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:11,324 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:11,324 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:11,324 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:11,324 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:11,325 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,325 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:11,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,325 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:11,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:11,325 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:11,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,326 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,328 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,328 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,328 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:11,328 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,328 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:11,328 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:11,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:11,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:11,328 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:11,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:11,328 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,328 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,328 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,328 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:11,328 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:11,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:11,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:11,428 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:11,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:11,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:11,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:11,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:11,431 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:11,431 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:11,431 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:11,431 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:11,432 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,432 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,432 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:11,432 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:11,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:11,433 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,433 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:11,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:11,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:11,433 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:11,433 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:11,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:11,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:11,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:11,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:11,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:11,434 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:11,434 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:11,434 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:11,434 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:11,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:11,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:11,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:11,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:11,435 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,435 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:11,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:11,435 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,435 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:11,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:11,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:11,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:11,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:11,436 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:11,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@488cc880[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:11,436 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,437 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,437 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:11,437 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:11,437 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:11,437 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,437 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:11,438 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,438 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:11,438 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:11,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:11,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:11,438 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:11,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:11,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:11,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:11,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:11,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:11,440 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:11,440 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:11,440 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:11,440 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:11,440 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:11,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:11,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:11,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:11,441 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:11,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:11,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:11,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:11,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,442 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:11,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,443 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,443 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:11,443 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:11,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:11,443 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,443 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:11,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:11,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:11,443 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,443 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:11,444 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:11,444 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,444 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:11,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:11,444 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:11,444 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:11,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:11,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56226 deadline: 1679273831521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,544 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:11,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:11,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:11,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:11,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:11,547 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:11,547 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:11,547 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:11,547 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:11,548 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,548 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,548 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:11,548 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:11,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:11,549 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,549 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:11,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:11,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:11,549 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:11,549 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:11,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:11,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:11,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:11,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:11,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:11,550 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:11,550 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:11,550 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:11,550 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:11,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:11,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:11,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:11,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:11,551 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,551 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:11,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:11,551 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:11,551 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:11,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:11,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:11,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:11,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ec4e478[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:11,568 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:11,568 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,569 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,569 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:11,569 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:11,569 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:11,569 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,569 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:11,570 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,570 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:11,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:11,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:11,570 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:11,570 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:11,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:11,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:11,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:11,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:11,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:11,572 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:11,572 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:11,572 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:11,572 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:11,572 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:11,573 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,573 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:11,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:11,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,573 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:11,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:11,574 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:11,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:11,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:11,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,575 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:11,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,578 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,578 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,578 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,578 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:11,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:11,578 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:11,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:11,578 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:11,579 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:11,579 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,579 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,579 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,579 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:11,579 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:11,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:11,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:11,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:11,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:11,587 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=236 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/dd53309d29e64eb39c9f6be1333f5d43 2023-03-20 00:56:11,679 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:11,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:11,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:11,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:11,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:11,682 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:11,682 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:11,682 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:11,682 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:11,683 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,683 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,683 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:11,683 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:11,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:11,684 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,684 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:11,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:11,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,684 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:11,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:11,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:11,685 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:11,685 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:11,685 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:11,685 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:11,685 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:11,685 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:11,685 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:11,685 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:11,685 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:11,685 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:11,685 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:11,685 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,685 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:11,685 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:11,686 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,686 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:11,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:11,686 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,686 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:11,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:11,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:11,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:11,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:11,687 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:11,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f6bee57[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:11,687 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,688 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,688 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:11,688 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,688 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:11,688 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:11,688 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:11,689 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,689 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:11,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:11,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:11,689 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:11,689 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:11,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:11,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:11,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:11,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:11,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:11,690 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:11,690 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:11,691 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:11,691 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:11,691 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:11,691 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,691 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,691 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:11,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:11,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:11,692 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:11,693 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:11,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,710 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:11,710 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:11,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:11,711 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:11,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:11,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:11,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:11,711 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,711 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:11,711 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:11,711 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,711 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:11,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:11,711 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:11,711 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,711 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,711 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,811 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:11,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:11,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:11,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:11,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:11,819 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:11,819 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:11,819 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:11,819 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:11,819 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,819 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,820 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:11,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:11,820 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:11,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,820 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,820 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:11,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:11,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:11,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:11,821 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:11,821 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:11,821 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:11,822 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:11,822 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:11,822 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:11,822 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:11,822 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:11,822 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:11,822 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:11,822 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:11,822 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:11,822 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,822 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:11,822 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:11,823 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:11,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:11,824 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,824 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,824 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:11,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:11,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:11,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:11,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:11,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41cda7c2[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:11,827 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:11,827 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,828 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,828 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,828 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:11,828 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:11,828 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:11,828 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:11,829 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,829 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:11,829 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:11,829 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:11,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:11,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:11,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:11,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:11,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:11,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:11,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:11,833 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:11,833 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:11,833 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:11,833 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:11,833 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:11,835 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,835 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:11,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,836 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:11,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:11,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:11,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:11,837 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:11,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:11,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,840 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:11,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,850 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:11,850 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:11,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:11,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:11,850 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,850 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,850 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,850 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,850 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,850 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,850 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,851 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:11,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:11,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:11,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:11,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:11,851 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:11,851 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:11,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:11,951 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:11,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:11,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:11,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:11,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:11,954 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:11,954 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:11,954 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:11,954 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:11,954 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,954 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,955 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:11,955 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:11,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:11,955 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,955 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:11,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:11,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,956 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:11,956 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:11,956 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:11,956 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:11,956 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:11,956 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:11,956 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:11,956 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:11,956 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:11,956 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:11,956 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:11,957 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:11,957 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:11,957 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:11,957 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,957 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:11,957 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:11,957 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,957 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:11,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:11,958 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,958 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:11,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:11,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:11,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:11,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:11,959 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:11,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f2bafb3[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:11,959 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,959 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,959 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:11,959 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:11,959 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:11,960 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:11,960 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,960 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:11,960 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:11,960 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:11,960 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:11,960 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:11,960 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:11,960 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:11,961 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:11,961 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:11,961 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,961 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:11,961 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:11,962 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,963 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:11,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,964 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,964 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,964 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:11,964 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:11,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:11,964 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:11,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:11,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:11,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:11,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:11,964 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:11,964 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:11,965 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:11,965 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:11,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:11,965 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:11,965 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:11,965 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:11,999 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=236 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/139b2e9894ba476ca375bb9c8a6bd448 2023-03-20 00:56:12,004 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/3b5295f32bbc46099a6c416648abeeb6 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/3b5295f32bbc46099a6c416648abeeb6 2023-03-20 00:56:12,009 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/3b5295f32bbc46099a6c416648abeeb6, entries=200, sequenceid=236, filesize=32.1 K 2023-03-20 00:56:12,010 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/dd53309d29e64eb39c9f6be1333f5d43 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/dd53309d29e64eb39c9f6be1333f5d43 2023-03-20 00:56:12,014 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/dd53309d29e64eb39c9f6be1333f5d43, entries=150, sequenceid=236, filesize=11.8 K 2023-03-20 00:56:12,015 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/139b2e9894ba476ca375bb9c8a6bd448 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/139b2e9894ba476ca375bb9c8a6bd448 2023-03-20 00:56:12,020 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/139b2e9894ba476ca375bb9c8a6bd448, entries=150, sequenceid=236, filesize=11.8 K 2023-03-20 00:56:12,021 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for f06a3c70774f5757abec4505bc3cdf91 in 1680ms, sequenceid=236, compaction requested=true 2023-03-20 00:56:12,021 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:12,021 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-03-20 00:56:12,021 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:12,021 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:12,022 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:12,022 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-03-20 00:56:12,027 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 239960 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-03-20 00:56:12,028 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): f06a3c70774f5757abec4505bc3cdf91/A is initiating minor compaction (all files) 2023-03-20 00:56:12,028 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f06a3c70774f5757abec4505bc3cdf91/A in TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:12,028 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 108230 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-03-20 00:56:12,028 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): f06a3c70774f5757abec4505bc3cdf91/B is initiating minor compaction (all files) 2023-03-20 00:56:12,028 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/71fa428390b14b0b95c14b139dc1a595, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/c5f782e1394b45238809df99261b49a4, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/8c669a1198e143f4a63435beb9880e6e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/5fb70bbd110a43e6b3347fd64598fb8c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/87fd9a4aa80b4af695bd530873ac9d13, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/7cd8c20e3c4d4da2b4057fab3eea61d3, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/726552578d4546bb83d305563cb29167, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/77cb88c105f44de18ff0e9e3bfad35b8, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/3b5295f32bbc46099a6c416648abeeb6] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp, totalSize=234.3 K 2023-03-20 00:56:12,028 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f06a3c70774f5757abec4505bc3cdf91/B in TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:12,028 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/2d80ed2d849d4b7991ef129825b9647d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/ddcf88331ff14d68a63d62ef4286644b, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/8ed0bcc71f1f410bbaa8d3e11188501d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/c2ade211bd694abebacaedeac504a3ec, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/8a09574f74f64107a70e91a1505512b6, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/940e7d64c004464882f4ab983c21f277, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/377a137bb2554784a942f581826b358b, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/9d39977c9f264cc08028f3c8e8e1e30c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/dd53309d29e64eb39c9f6be1333f5d43] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp, totalSize=105.7 K 2023-03-20 00:56:12,028 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 71fa428390b14b0b95c14b139dc1a595, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1679273761003 2023-03-20 00:56:12,029 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 2d80ed2d849d4b7991ef129825b9647d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1679273761003 2023-03-20 00:56:12,029 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting c5f782e1394b45238809df99261b49a4, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1679273761163 2023-03-20 00:56:12,029 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting ddcf88331ff14d68a63d62ef4286644b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1679273761163 2023-03-20 00:56:12,029 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 8c669a1198e143f4a63435beb9880e6e, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1679273762305 2023-03-20 00:56:12,030 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 8ed0bcc71f1f410bbaa8d3e11188501d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1679273762305 2023-03-20 00:56:12,030 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting c2ade211bd694abebacaedeac504a3ec, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1679273762938 2023-03-20 00:56:12,030 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 5fb70bbd110a43e6b3347fd64598fb8c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1679273762938 2023-03-20 00:56:12,031 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 8a09574f74f64107a70e91a1505512b6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1679273764077 2023-03-20 00:56:12,031 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 87fd9a4aa80b4af695bd530873ac9d13, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1679273764077 2023-03-20 00:56:12,032 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 940e7d64c004464882f4ab983c21f277, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1679273764710 2023-03-20 00:56:12,032 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 7cd8c20e3c4d4da2b4057fab3eea61d3, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1679273764710 2023-03-20 00:56:12,032 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 377a137bb2554784a942f581826b358b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1679273766361 2023-03-20 00:56:12,032 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 726552578d4546bb83d305563cb29167, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1679273766361 2023-03-20 00:56:12,032 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 9d39977c9f264cc08028f3c8e8e1e30c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1679273768540 2023-03-20 00:56:12,032 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 77cb88c105f44de18ff0e9e3bfad35b8, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1679273768540 2023-03-20 00:56:12,033 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting dd53309d29e64eb39c9f6be1333f5d43, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1679273769214 2023-03-20 00:56:12,033 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 3b5295f32bbc46099a6c416648abeeb6, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1679273769214 2023-03-20 00:56:12,051 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): f06a3c70774f5757abec4505bc3cdf91#B#compaction#144 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:12,059 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): f06a3c70774f5757abec4505bc3cdf91#A#compaction#145 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:12,065 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:12,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:12,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:12,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:12,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:12,067 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:12,068 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:12,068 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:12,068 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:12,068 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:12,068 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:12,070 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:12,070 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:12,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:12,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:12,070 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:12,070 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:12,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:12,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:12,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:12,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:12,071 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:12,071 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:12,071 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:12,071 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:12,071 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:12,071 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:12,071 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:12,071 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:12,071 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:12,071 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:12,072 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f06a3c70774f5757abec4505bc3cdf91 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-20 00:56:12,072 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=A 2023-03-20 00:56:12,072 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:12,072 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=B 2023-03-20 00:56:12,072 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:12,072 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=C 2023-03-20 00:56:12,072 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:12,467 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/d278b1a3082a4d51ac5db87a21adfc1f as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/d278b1a3082a4d51ac5db87a21adfc1f 2023-03-20 00:56:12,472 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in f06a3c70774f5757abec4505bc3cdf91/B of f06a3c70774f5757abec4505bc3cdf91 into d278b1a3082a4d51ac5db87a21adfc1f(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:12,472 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:12,473 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91., storeName=f06a3c70774f5757abec4505bc3cdf91/B, priority=7, startTime=1679273772021; duration=0sec 2023-03-20 00:56:12,473 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:12,473 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-03-20 00:56:12,476 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 108230 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-03-20 00:56:12,477 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): f06a3c70774f5757abec4505bc3cdf91/C is initiating minor compaction (all files) 2023-03-20 00:56:12,477 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f06a3c70774f5757abec4505bc3cdf91/C in TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:12,477 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/95de81ae673b4ee987c1af7925ef1d08, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/e9d4c3cd33af444f87b72e8859f6c10a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/8423f8f8a7a54f248a6d705ce555f65d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/063f2a5c739047d2bb64b92e3b89aca2, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/54a09d3700614249b1533c622787cfcb, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/c1d17a093eeb440e8a99f00af145f8bc, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/24741bbcd5e2464a8f9705c08137ac65, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/382486d354594cb5b71fd0cb7b97da9a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/139b2e9894ba476ca375bb9c8a6bd448] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp, totalSize=105.7 K 2023-03-20 00:56:12,477 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 95de81ae673b4ee987c1af7925ef1d08, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1679273761003 2023-03-20 00:56:12,478 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting e9d4c3cd33af444f87b72e8859f6c10a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1679273761163 2023-03-20 00:56:12,478 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 8423f8f8a7a54f248a6d705ce555f65d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1679273762305 2023-03-20 00:56:12,478 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 063f2a5c739047d2bb64b92e3b89aca2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1679273762938 2023-03-20 00:56:12,479 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 54a09d3700614249b1533c622787cfcb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1679273764077 2023-03-20 00:56:12,479 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting c1d17a093eeb440e8a99f00af145f8bc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1679273764710 2023-03-20 00:56:12,479 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 24741bbcd5e2464a8f9705c08137ac65, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1679273766361 2023-03-20 00:56:12,480 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 382486d354594cb5b71fd0cb7b97da9a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1679273768540 2023-03-20 00:56:12,480 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 139b2e9894ba476ca375bb9c8a6bd448, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1679273769214 2023-03-20 00:56:12,507 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032020b0d4dc308e46eb94d952d3ce01beac to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032020b0d4dc308e46eb94d952d3ce01beac 2023-03-20 00:56:12,509 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): f06a3c70774f5757abec4505bc3cdf91#C#compaction#147 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:12,537 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:12,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:56:12,546 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=251, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/6287161e224b48a0b02eded421d3792e 2023-03-20 00:56:12,588 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/c56768c7101d47d7985482cceef96a97 2023-03-20 00:56:12,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:12,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56226 deadline: 1679273832634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:12,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:12,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56226 deadline: 1679273832737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:12,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:12,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56218 deadline: 1679273832873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:12,876 DEBUG [Thread-763] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8164 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91., hostname=jenkins-hbase20.apache.org,34553,1679273725199, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 00:56:12,885 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/2ad4f74448a140cfa8a6c642e29beef6 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/2ad4f74448a140cfa8a6c642e29beef6 2023-03-20 00:56:12,886 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:12,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56200 deadline: 1679273832884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:12,888 DEBUG [Thread-771] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8171 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91., hostname=jenkins-hbase20.apache.org,34553,1679273725199, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 00:56:12,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:12,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56176 deadline: 1679273832889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:12,894 DEBUG [Thread-769] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8181 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91., hostname=jenkins-hbase20.apache.org,34553,1679273725199, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 00:56:12,897 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in f06a3c70774f5757abec4505bc3cdf91/A of f06a3c70774f5757abec4505bc3cdf91 into 2ad4f74448a140cfa8a6c642e29beef6(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:12,897 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:12,897 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91., storeName=f06a3c70774f5757abec4505bc3cdf91/A, priority=7, startTime=1679273772021; duration=0sec 2023-03-20 00:56:12,897 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:12,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:12,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:56216 deadline: 1679273832898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:12,905 DEBUG [Thread-767] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8190 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91., hostname=jenkins-hbase20.apache.org,34553,1679273725199, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 00:56:12,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:12,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:56226 deadline: 1679273832941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:12,962 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/9b28273df1a84012b8c698a8ae374a45 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/9b28273df1a84012b8c698a8ae374a45 2023-03-20 00:56:12,978 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in f06a3c70774f5757abec4505bc3cdf91/C of f06a3c70774f5757abec4505bc3cdf91 into 9b28273df1a84012b8c698a8ae374a45(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:12,978 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:12,978 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91., storeName=f06a3c70774f5757abec4505bc3cdf91/C, priority=7, startTime=1679273772022; duration=0sec 2023-03-20 00:56:12,978 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:13,004 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/ee10cac1ba0746a08c019041f2a203f2 2023-03-20 00:56:13,012 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/6287161e224b48a0b02eded421d3792e as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/6287161e224b48a0b02eded421d3792e 2023-03-20 00:56:13,019 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/6287161e224b48a0b02eded421d3792e, entries=150, sequenceid=251, filesize=25.3 K 2023-03-20 00:56:13,023 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/c56768c7101d47d7985482cceef96a97 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/c56768c7101d47d7985482cceef96a97 2023-03-20 00:56:13,036 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/c56768c7101d47d7985482cceef96a97, entries=150, sequenceid=251, filesize=11.8 K 2023-03-20 00:56:13,037 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/ee10cac1ba0746a08c019041f2a203f2 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/ee10cac1ba0746a08c019041f2a203f2 2023-03-20 00:56:13,042 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/ee10cac1ba0746a08c019041f2a203f2, entries=150, sequenceid=251, filesize=11.8 K 2023-03-20 00:56:13,044 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for f06a3c70774f5757abec4505bc3cdf91 in 972ms, sequenceid=251, compaction requested=false 2023-03-20 00:56:13,044 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:13,044 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:13,044 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:13,044 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:13,044 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:13,044 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:13,044 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:13,045 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:13,045 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:13,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:13,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:13,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:13,045 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:13,045 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:13,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:13,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4665610a[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:13,047 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:13,047 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:13,047 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:13,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:13,047 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:13,047 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:13,047 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:13,047 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:13,047 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:13,048 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:13,048 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:13,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:13,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:13,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:13,048 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:13,048 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:13,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:13,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:13,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:13,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:13,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:13,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:13,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:13,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:13,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:13,050 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:13,050 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:13,050 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:13,051 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:13,051 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:13,051 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:13,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:13,052 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:13,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:13,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:13,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:13,052 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:13,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:13,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:13,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:13,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:13,052 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:13,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:13,054 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:13,056 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:13,056 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:13,056 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:13,056 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:13,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:13,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:13,056 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:13,056 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:13,056 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:13,056 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:13,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:13,056 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:13,056 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:13,056 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:13,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:13,056 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:13,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:13,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:13,157 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:13,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:13,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:13,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:13,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:13,161 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:13,161 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:13,161 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:13,161 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:13,161 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:13,161 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:13,162 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:13,162 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:13,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:13,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:13,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:13,162 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:13,162 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:13,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:13,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:13,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:13,163 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:13,163 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:13,163 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:13,163 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:13,163 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:13,163 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:13,166 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:13,166 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:13,166 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:13,166 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:13,166 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f06a3c70774f5757abec4505bc3cdf91 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-20 00:56:13,166 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=A 2023-03-20 00:56:13,166 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:13,167 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=B 2023-03-20 00:56:13,167 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:13,167 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=C 2023-03-20 00:56:13,167 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:13,198 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320b23f54e965b3474e8b656b1299b41a53 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320b23f54e965b3474e8b656b1299b41a53 2023-03-20 00:56:13,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:56:13,247 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:13,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:13,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:56226 deadline: 1679273833273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:13,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:13,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:56226 deadline: 1679273833375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:13,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:13,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:56226 deadline: 1679273833578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:13,609 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=271, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/ac2a045374914aec924493af4a18a8c9 2023-03-20 00:56:13,625 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/48c80e1306c246549d1eade290fa18f0 2023-03-20 00:56:13,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:13,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:56226 deadline: 1679273833880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:14,036 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/57f537bf126f47d5aa1c3c827f32f700 2023-03-20 00:56:14,044 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/ac2a045374914aec924493af4a18a8c9 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/ac2a045374914aec924493af4a18a8c9 2023-03-20 00:56:14,052 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/ac2a045374914aec924493af4a18a8c9, entries=150, sequenceid=271, filesize=25.4 K 2023-03-20 00:56:14,053 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/48c80e1306c246549d1eade290fa18f0 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/48c80e1306c246549d1eade290fa18f0 2023-03-20 00:56:14,061 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/48c80e1306c246549d1eade290fa18f0, entries=150, sequenceid=271, filesize=11.9 K 2023-03-20 00:56:14,062 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/57f537bf126f47d5aa1c3c827f32f700 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/57f537bf126f47d5aa1c3c827f32f700 2023-03-20 00:56:14,077 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/57f537bf126f47d5aa1c3c827f32f700, entries=150, sequenceid=271, filesize=11.9 K 2023-03-20 00:56:14,079 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for f06a3c70774f5757abec4505bc3cdf91 in 913ms, sequenceid=271, compaction requested=true 2023-03-20 00:56:14,079 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:14,079 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:14,080 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:14,080 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:14,080 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:14,080 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:14,080 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:14,081 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:14,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:14,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:14,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:14,081 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:14,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:14,081 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:14,081 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:14,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:14,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:14,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:14,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:14,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:14,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@269c36c7[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:14,083 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:14,083 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:14,084 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:14,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:14,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:14,085 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:14,085 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:14,085 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:14,085 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:14,085 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:14,086 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:14,086 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:14,086 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:14,086 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:14,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:14,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:14,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:14,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:14,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:14,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:14,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:14,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:14,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:14,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:14,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:14,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:14,089 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:14,090 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:14,090 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:14,090 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:14,090 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:14,092 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:14,092 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:14,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:14,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:14,092 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:14,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:14,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:14,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:14,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:14,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:14,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:14,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:14,094 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:14,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:14,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:14,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:14,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:14,096 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:14,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:14,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:14,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:14,100 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:14,100 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:14,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:14,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:14,100 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:14,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:14,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:14,100 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:14,101 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:14,101 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:14,101 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:14,101 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:14,101 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:14,101 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:14,101 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:14,101 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:14,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:14,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:14,201 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:14,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:14,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:14,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:14,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:14,205 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:14,205 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:14,205 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:14,205 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:14,205 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:14,205 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:14,206 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:14,206 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:14,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:14,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:14,206 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:14,206 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:14,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:14,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:14,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:14,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:14,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:14,207 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:14,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:14,207 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:14,207 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:14,207 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:14,207 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:14,207 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:14,207 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:14,207 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:14,208 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f06a3c70774f5757abec4505bc3cdf91 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-20 00:56:14,208 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=A 2023-03-20 00:56:14,208 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:14,208 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=B 2023-03-20 00:56:14,208 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:14,208 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=C 2023-03-20 00:56:14,208 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:14,223 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320c68fa760a5b74628b673406711cf0c88 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320c68fa760a5b74628b673406711cf0c88 2023-03-20 00:56:14,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:56:14,386 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:14,425 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:14,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:56226 deadline: 1679273834423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:14,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:14,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:56226 deadline: 1679273834526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:14,637 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=286, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/e3d8297d602247559770b9464dcb912a 2023-03-20 00:56:14,731 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:14,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:56226 deadline: 1679273834731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:15,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:56226 deadline: 1679273835032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,049 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/518a13b594d94818b116a46bb24feff2 2023-03-20 00:56:15,472 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/d6dcb9ef94184e16b569e72b1bc2ea2a 2023-03-20 00:56:15,488 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/e3d8297d602247559770b9464dcb912a as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/e3d8297d602247559770b9464dcb912a 2023-03-20 00:56:15,494 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/e3d8297d602247559770b9464dcb912a, entries=150, sequenceid=286, filesize=25.4 K 2023-03-20 00:56:15,495 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/518a13b594d94818b116a46bb24feff2 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/518a13b594d94818b116a46bb24feff2 2023-03-20 00:56:15,506 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/518a13b594d94818b116a46bb24feff2, entries=150, sequenceid=286, filesize=11.9 K 2023-03-20 00:56:15,508 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/d6dcb9ef94184e16b569e72b1bc2ea2a as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/d6dcb9ef94184e16b569e72b1bc2ea2a 2023-03-20 00:56:15,512 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/d6dcb9ef94184e16b569e72b1bc2ea2a, entries=150, sequenceid=286, filesize=11.9 K 2023-03-20 00:56:15,513 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for f06a3c70774f5757abec4505bc3cdf91 in 1305ms, sequenceid=286, compaction requested=true 2023-03-20 00:56:15,513 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:15,513 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:15,513 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:15,514 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:15,514 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,514 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:15,514 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:15,515 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,515 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:15,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:15,515 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,515 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:15,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:15,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:15,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:15,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:15,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:15,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a437661[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:15,516 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:15,516 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,517 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,517 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:15,517 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:15,517 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:15,517 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,517 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:15,518 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:15,518 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,518 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:15,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:15,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:15,518 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:15,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:15,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:15,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:15,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:15,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:15,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:15,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:15,529 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:15,529 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:15,529 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:15,529 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:15,529 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:15,535 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,535 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:15,535 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:15,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:15,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:15,536 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:15,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:15,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:15,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:15,537 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:15,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:15,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:15,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:56:15,541 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f06a3c70774f5757abec4505bc3cdf91 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-20 00:56:15,541 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=A 2023-03-20 00:56:15,541 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:15,541 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=B 2023-03-20 00:56:15,541 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:15,541 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=C 2023-03-20 00:56:15,541 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:15,542 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,542 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:15,542 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:15,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:15,542 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:15,542 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:15,542 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:15,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:15,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:15,542 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,542 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,542 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,542 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:15,542 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:15,542 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:15,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:15,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:15,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:15,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:15,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:56226 deadline: 1679273835578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,600 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303201e9eda4131b040ec87ad1c4a151855aa to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303201e9eda4131b040ec87ad1c4a151855aa 2023-03-20 00:56:15,615 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=304, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/ec6b308949314e829bdb37bc57bcbc9d 2023-03-20 00:56:15,643 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:15,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:15,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:15,650 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/5f04d41e5cd74add9a427a38dd0eca13 2023-03-20 00:56:15,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:15,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:15,653 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:15,653 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:15,653 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:15,653 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:15,654 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,654 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:15,655 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:15,655 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:15,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:15,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:15,656 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,656 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:15,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:15,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:15,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:15,666 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:15,666 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:15,666 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:15,666 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:15,666 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:15,675 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:15,675 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:15,675 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:15,675 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:15,675 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:15,675 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:15,675 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:15,675 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:15,675 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,678 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:15,678 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:15,679 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,679 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:15,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:15,680 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,680 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:15,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:15,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:15,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:15,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:15,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:15,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@382716ad[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:15,681 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:15,681 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,684 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,684 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,684 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:15,684 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:15,684 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:15,684 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:15,685 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:15,685 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:15,685 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:15,685 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:15,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:15,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:15,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:15,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:15,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:15,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:15,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:15,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:15,688 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:15,688 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:15,688 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:15,689 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:15,689 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:15,690 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,690 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:15,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,690 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:15,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:15,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:15,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:15,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:15,692 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:15,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:56226 deadline: 1679273835683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:15,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:15,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:15,694 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:15,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:15,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,696 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:15,696 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:15,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:15,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:15,696 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,696 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:15,696 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:15,696 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:15,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:15,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:15,696 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,697 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,697 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,697 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,697 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:15,697 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:15,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:15,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:15,797 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:15,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:15,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:15,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:15,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:15,800 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:15,800 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:15,801 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:15,801 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:15,801 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,801 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:15,802 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:15,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:15,802 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:15,802 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,802 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:15,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:15,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:15,803 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:15,803 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:15,803 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:15,803 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:15,803 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:15,803 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:15,803 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:15,803 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:15,803 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:15,804 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:15,804 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:15,804 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:15,804 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:15,804 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,804 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:15,804 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:15,805 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,805 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:15,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:15,805 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,805 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:15,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:15,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:15,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:15,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:15,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:15,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55a74d92[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:15,807 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:15,807 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,807 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,807 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:15,807 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:15,807 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:15,809 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,809 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:15,819 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:15,819 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,819 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:15,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,819 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:15,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:15,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:15,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:15,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:15,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:15,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:15,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:15,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:15,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:15,824 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:15,824 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:15,824 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:15,824 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:15,824 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:15,827 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:15,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:15,828 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,828 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:15,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:15,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,828 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:15,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:15,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:15,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:15,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:15,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,831 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:15,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:15,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,835 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:15,835 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,835 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:15,836 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:15,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:15,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:15,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:15,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:15,836 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:15,836 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:15,836 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:15,836 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:15,836 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,836 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,836 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,836 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:15,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:15,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:15,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:15,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:56226 deadline: 1679273835894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,936 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:15,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:15,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:15,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:15,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:15,941 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:15,941 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:15,941 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:15,941 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:15,941 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,941 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:15,942 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:15,942 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:15,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:15,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:15,943 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,943 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:15,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:15,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:15,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:15,944 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:15,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:15,944 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:15,944 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:15,944 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:15,944 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:15,944 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:15,944 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:15,944 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:15,945 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:15,945 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:15,945 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:15,945 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:15,945 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,945 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:15,945 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:15,946 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:15,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:15,946 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:15,947 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,947 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:15,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:15,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:15,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:15,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:15,948 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:15,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@124a73ef[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:15,948 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,949 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,949 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:15,949 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:15,949 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:15,952 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,952 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:15,952 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,952 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:15,952 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:15,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:15,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:15,952 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:15,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:15,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:15,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:15,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:15,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:15,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:15,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:15,956 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:15,956 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:15,956 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:15,956 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:15,957 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:15,958 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,958 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:15,958 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,959 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,960 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,962 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:15,962 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:15,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:15,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:15,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:15,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:15,963 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,963 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:15,963 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:15,963 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:15,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:15,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:15,964 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:15,964 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:15,963 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:15,963 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,964 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:15,964 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:15,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:15,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:15,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:15,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:15,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:15,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:16,064 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:16,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:16,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:16,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:16,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:16,070 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:16,070 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:16,070 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:16,070 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:16,070 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,070 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,071 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:16,071 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:16,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:16,074 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,074 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:16,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,075 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:16,075 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:16,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:16,075 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:16,075 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:16,075 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:16,078 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:16,078 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:16,078 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:16,078 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:16,078 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:16,078 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:16,078 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:16,078 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:16,078 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,078 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:16,078 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:16,087 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:16,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:16,087 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:16,088 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,088 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:16,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:16,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:16,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:16,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3abffb9[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:16,090 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:16,090 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,091 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,091 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:16,091 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,091 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:16,091 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:16,091 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:16,093 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,093 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:16,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:16,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:16,093 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:16,093 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:16,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:16,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:16,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:16,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:16,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:16,095 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:16,095 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:16,095 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:16,095 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:16,095 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:16,097 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:16,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:16,097 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,097 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:16,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:16,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:16,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:16,097 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:16,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,099 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:16,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,100 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/218b9184bf8349c7a2a79ddfac4e7e1e 2023-03-20 00:56:16,101 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:16,101 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:16,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:16,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:16,101 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,101 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,101 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,101 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,101 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,101 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,101 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:16,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:16,101 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:16,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:16,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:16,102 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:16,102 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:16,106 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/ec6b308949314e829bdb37bc57bcbc9d as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/ec6b308949314e829bdb37bc57bcbc9d 2023-03-20 00:56:16,112 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/ec6b308949314e829bdb37bc57bcbc9d, entries=150, sequenceid=304, filesize=25.4 K 2023-03-20 00:56:16,113 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/5f04d41e5cd74add9a427a38dd0eca13 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/5f04d41e5cd74add9a427a38dd0eca13 2023-03-20 00:56:16,122 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/5f04d41e5cd74add9a427a38dd0eca13, entries=150, sequenceid=304, filesize=11.9 K 2023-03-20 00:56:16,123 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/218b9184bf8349c7a2a79ddfac4e7e1e as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/218b9184bf8349c7a2a79ddfac4e7e1e 2023-03-20 00:56:16,133 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/218b9184bf8349c7a2a79ddfac4e7e1e, entries=150, sequenceid=304, filesize=11.9 K 2023-03-20 00:56:16,133 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for f06a3c70774f5757abec4505bc3cdf91 in 592ms, sequenceid=304, compaction requested=true 2023-03-20 00:56:16,134 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:16,134 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:16,134 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-20 00:56:16,134 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-20 00:56:16,134 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:16,134 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:16,138 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61142 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-20 00:56:16,138 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 130442 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-20 00:56:16,138 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): f06a3c70774f5757abec4505bc3cdf91/B is initiating minor compaction (all files) 2023-03-20 00:56:16,138 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): f06a3c70774f5757abec4505bc3cdf91/A is initiating minor compaction (all files) 2023-03-20 00:56:16,138 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f06a3c70774f5757abec4505bc3cdf91/B in TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:16,138 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f06a3c70774f5757abec4505bc3cdf91/A in TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:16,138 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/2ad4f74448a140cfa8a6c642e29beef6, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/6287161e224b48a0b02eded421d3792e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/ac2a045374914aec924493af4a18a8c9, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/e3d8297d602247559770b9464dcb912a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/ec6b308949314e829bdb37bc57bcbc9d] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp, totalSize=127.4 K 2023-03-20 00:56:16,138 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/d278b1a3082a4d51ac5db87a21adfc1f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/c56768c7101d47d7985482cceef96a97, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/48c80e1306c246549d1eade290fa18f0, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/518a13b594d94818b116a46bb24feff2, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/5f04d41e5cd74add9a427a38dd0eca13] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp, totalSize=59.7 K 2023-03-20 00:56:16,139 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting d278b1a3082a4d51ac5db87a21adfc1f, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1679273769214 2023-03-20 00:56:16,139 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 2ad4f74448a140cfa8a6c642e29beef6, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1679273769214 2023-03-20 00:56:16,139 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting c56768c7101d47d7985482cceef96a97, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1679273770372 2023-03-20 00:56:16,139 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 6287161e224b48a0b02eded421d3792e, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1679273770372 2023-03-20 00:56:16,139 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 48c80e1306c246549d1eade290fa18f0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1679273772617 2023-03-20 00:56:16,140 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting ac2a045374914aec924493af4a18a8c9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1679273772617 2023-03-20 00:56:16,140 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 518a13b594d94818b116a46bb24feff2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1679273773255 2023-03-20 00:56:16,140 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting e3d8297d602247559770b9464dcb912a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1679273773255 2023-03-20 00:56:16,140 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 5f04d41e5cd74add9a427a38dd0eca13, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1679273774400 2023-03-20 00:56:16,140 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting ec6b308949314e829bdb37bc57bcbc9d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1679273774400 2023-03-20 00:56:16,152 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): f06a3c70774f5757abec4505bc3cdf91#B#compaction#160 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:16,180 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): f06a3c70774f5757abec4505bc3cdf91#A#compaction#159 average throughput is 0.66 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:16,193 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/7d3f0e669cde4237b0f4d62c6ebbbd48 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/7d3f0e669cde4237b0f4d62c6ebbbd48 2023-03-20 00:56:16,202 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:16,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:16,204 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in f06a3c70774f5757abec4505bc3cdf91/B of f06a3c70774f5757abec4505bc3cdf91 into 7d3f0e669cde4237b0f4d62c6ebbbd48(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:16,204 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:16,204 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91., storeName=f06a3c70774f5757abec4505bc3cdf91/B, priority=11, startTime=1679273776134; duration=0sec 2023-03-20 00:56:16,204 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:16,204 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-20 00:56:16,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:16,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:56:16,207 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f06a3c70774f5757abec4505bc3cdf91 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-20 00:56:16,207 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61142 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-20 00:56:16,207 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): f06a3c70774f5757abec4505bc3cdf91/C is initiating minor compaction (all files) 2023-03-20 00:56:16,207 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f06a3c70774f5757abec4505bc3cdf91/C in TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:16,207 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/9b28273df1a84012b8c698a8ae374a45, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/ee10cac1ba0746a08c019041f2a203f2, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/57f537bf126f47d5aa1c3c827f32f700, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/d6dcb9ef94184e16b569e72b1bc2ea2a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/218b9184bf8349c7a2a79ddfac4e7e1e] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp, totalSize=59.7 K 2023-03-20 00:56:16,207 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=A 2023-03-20 00:56:16,207 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:16,208 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=B 2023-03-20 00:56:16,208 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:16,208 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=C 2023-03-20 00:56:16,208 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:16,208 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 9b28273df1a84012b8c698a8ae374a45, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1679273769214 2023-03-20 00:56:16,208 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting ee10cac1ba0746a08c019041f2a203f2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1679273770372 2023-03-20 00:56:16,209 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 57f537bf126f47d5aa1c3c827f32f700, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1679273772617 2023-03-20 00:56:16,209 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting d6dcb9ef94184e16b569e72b1bc2ea2a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1679273773255 2023-03-20 00:56:16,210 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 218b9184bf8349c7a2a79ddfac4e7e1e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1679273774400 2023-03-20 00:56:16,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:16,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:16,218 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:16,218 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:16,218 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:16,218 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:16,218 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,219 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,219 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,219 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:16,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:16,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:16,220 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,220 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:16,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:16,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:16,228 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:16,228 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:16,228 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:16,228 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:16,229 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:16,238 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:16,238 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:16,238 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:16,238 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:16,238 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:16,238 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:16,238 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:16,238 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:16,238 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,244 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): f06a3c70774f5757abec4505bc3cdf91#C#compaction#161 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:16,245 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:16,245 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:16,246 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,246 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:16,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:16,247 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,247 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:16,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:16,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:16,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:16,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:16,248 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:16,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@339131f[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:16,248 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,249 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,249 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:16,249 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:16,249 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:16,249 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,249 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:16,250 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,250 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:16,250 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:16,250 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:16,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:16,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:16,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:16,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:16,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:16,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:16,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:16,255 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:16,255 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:16,255 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:16,255 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:16,255 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:16,260 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,260 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:16,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,261 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:16,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:16,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:16,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:16,261 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:16,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:16,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,262 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:16,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,267 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,267 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,267 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,267 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,267 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,267 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:16,267 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:16,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:16,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:16,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:16,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:16,267 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:16,267 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:16,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:56226 deadline: 1679273836265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:16,267 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,267 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:16,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:16,267 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:16,268 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:16,274 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/13a9b92c731e401aaffc531a8cbab4e8 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/13a9b92c731e401aaffc531a8cbab4e8 2023-03-20 00:56:16,279 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303200303eadf496e41dabfeee8dd5191fa33 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303200303eadf496e41dabfeee8dd5191fa33 2023-03-20 00:56:16,287 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in f06a3c70774f5757abec4505bc3cdf91/C of f06a3c70774f5757abec4505bc3cdf91 into 13a9b92c731e401aaffc531a8cbab4e8(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:16,288 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:16,288 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91., storeName=f06a3c70774f5757abec4505bc3cdf91/C, priority=11, startTime=1679273776134; duration=0sec 2023-03-20 00:56:16,288 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:16,368 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:16,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:16,369 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:16,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:56226 deadline: 1679273836369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:16,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:16,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:16,373 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:16,373 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:16,373 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:16,373 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:16,374 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,374 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,375 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,375 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:16,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:16,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:16,375 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,375 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:16,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:16,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:16,385 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:16,385 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:16,385 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:16,386 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:16,386 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:16,394 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:16,394 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:16,402 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:16,402 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:16,402 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:16,402 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:16,403 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:16,403 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:16,403 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,403 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:16,403 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:16,404 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,404 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:16,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:16,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:16,405 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,405 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:16,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:16,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:16,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:16,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c1d0537[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:16,407 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:16,407 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,408 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,408 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:16,408 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:16,408 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:16,408 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,408 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:16,409 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:16,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:16,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:16,410 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:16,410 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:16,410 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:16,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:16,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:16,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:16,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:16,412 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:16,412 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:16,412 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:16,412 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:16,412 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:16,414 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:16,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:16,414 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,414 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:16,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:16,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:16,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:16,417 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:16,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,419 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:16,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,427 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:16,427 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:16,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:16,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:16,428 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,428 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,428 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,428 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:16,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:16,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:16,428 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:16,428 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,428 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,428 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:16,428 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:16,428 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:16,528 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:16,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:16,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:16,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:16,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:16,538 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:16,538 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:16,538 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:16,538 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:16,539 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,539 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,539 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,539 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:16,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:16,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:16,540 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,540 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:16,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:16,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,541 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:16,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:16,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:16,542 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:16,542 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:16,542 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:16,556 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:16,556 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:16,556 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:16,556 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:16,557 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:16,557 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:16,557 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:16,557 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:16,557 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,557 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:16,557 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:16,574 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:16,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:56226 deadline: 1679273836571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,575 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,575 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:16,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:16,575 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,575 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:16,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:16,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:16,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:16,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:16,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d89c650[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:16,577 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:16,577 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,586 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,586 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,586 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:16,586 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:16,586 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:16,586 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:16,587 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:16,587 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:16,588 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:16,587 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:16,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:16,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:16,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:16,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:16,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:16,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:16,592 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:16,592 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:16,592 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:16,592 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:16,592 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:16,603 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,603 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,603 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:16,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:16,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:16,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:16,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:16,603 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:16,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:16,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,609 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:16,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,620 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,620 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,621 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,621 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,621 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,621 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,621 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,621 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:16,621 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:16,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:16,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:16,621 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:16,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:16,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:16,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:16,621 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:16,621 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:16,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:16,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:16,694 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=320, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/b2d9fb8e49174d54b0462b6179dd0c02 2023-03-20 00:56:16,721 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:16,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:16,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:16,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:16,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:16,725 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:16,725 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:16,725 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:16,725 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:16,725 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,725 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,726 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:16,726 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:16,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:16,726 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/07a9130d29d14684b29e95150f117ea6 2023-03-20 00:56:16,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,726 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,726 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:16,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,727 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:16,727 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:16,727 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:16,727 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:16,727 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:16,727 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:16,730 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:16,730 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:16,730 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:16,730 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:16,730 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:16,730 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:16,730 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:16,730 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:16,730 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,730 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:16,730 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:16,732 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:16,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:16,732 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,732 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:16,732 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:16,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:16,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:16,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:16,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7183b26f[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:16,734 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:16,734 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,735 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,735 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:16,735 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,735 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:16,735 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:16,735 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:16,738 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:16,738 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:16,738 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:16,738 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:16,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:16,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:16,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:16,740 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:16,740 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:16,740 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:16,740 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:16,740 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:16,760 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,760 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:16,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,760 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:16,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:16,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:16,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:16,761 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:16,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:16,762 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:16,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,764 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,764 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,764 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,764 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:16,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:16,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:16,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:16,764 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:16,764 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:16,764 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:16,764 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,764 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:16,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:16,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:16,764 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,766 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,766 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:16,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:16,864 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:16,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:16,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:16,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:16,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:16,867 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:16,867 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:16,867 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:16,867 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:16,868 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,868 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,868 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:16,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:16,868 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:16,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,869 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,869 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:16,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:16,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,869 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:16,869 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:16,869 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:16,869 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:16,870 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:16,870 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:16,870 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:16,870 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:16,870 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:16,870 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:16,870 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:16,870 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:16,870 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:16,870 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:16,870 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,870 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:16,870 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:16,871 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,871 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:16,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:16,871 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,871 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:16,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:16,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:16,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:16,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:16,872 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:16,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ae9e6ab[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:16,872 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,873 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,873 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:16,873 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:16,873 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:16,873 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,873 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:16,873 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,873 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:16,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:16,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:16,873 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:16,874 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:16,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:16,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:16,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:16,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:16,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:16,876 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:16,876 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:16,876 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:16,876 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:16,876 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:16,877 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,877 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:16,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:16,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:16,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,877 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,878 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:16,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:56226 deadline: 1679273836877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,879 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:16,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:16,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:16,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,880 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,880 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:16,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:16,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:16,880 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:16,880 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:16,880 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,880 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,881 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,881 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,881 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,881 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:16,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:16,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:16,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:16,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:16,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:16,881 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:16,881 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:16,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:16,981 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:16,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:16,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:16,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:16,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:16,985 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:16,985 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:16,985 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:16,985 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:16,985 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,985 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,987 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:16,987 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:16,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:16,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,988 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,988 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:16,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:16,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:16,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:16,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:16,989 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:16,989 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:16,989 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:16,989 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:16,989 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:16,990 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:16,990 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:16,990 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:16,990 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:16,990 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:16,990 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:16,990 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:16,990 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:16,990 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,990 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:16,990 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:16,991 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,991 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:16,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:16,992 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,992 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:16,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:16,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:16,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:16,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:16,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:16,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ed9687c[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:16,995 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:16,995 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,998 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:16,998 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:16,998 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:16,998 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:16,998 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:16,998 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:17,000 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:17,000 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:17,000 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:17,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:17,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:17,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:17,000 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:17,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:17,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:17,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:17,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:17,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:17,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:17,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:17,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:17,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:17,003 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:17,003 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:17,003 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:17,003 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:17,003 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:17,008 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:17,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:17,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:17,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:17,009 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:17,009 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:17,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:17,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:17,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:17,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:17,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:17,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:17,010 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:17,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:17,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:17,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:17,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:17,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:17,011 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:17,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:17,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:17,019 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:17,019 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:17,020 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:17,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:17,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:17,020 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:17,020 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:17,020 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:17,020 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:17,020 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:17,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:17,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:17,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:17,020 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:17,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:17,020 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:17,021 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:17,021 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:17,121 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:17,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:17,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:17,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:17,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:17,125 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:17,125 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:17,125 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:17,125 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:17,126 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:17,126 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:17,126 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:17,127 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:17,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:17,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:17,127 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:17,127 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:17,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:17,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:17,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:17,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:17,127 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:17,128 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:17,128 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:17,128 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:17,128 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:17,128 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:17,133 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:17,133 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:17,133 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:17,134 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:17,134 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:17,134 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:17,134 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:17,134 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:17,134 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:17,134 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:17,134 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:17,135 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:17,135 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:17,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:17,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:17,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:17,135 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:17,135 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:17,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:17,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:17,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:17,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:17,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:17,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:17,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d8192b1[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:17,136 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:17,136 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:17,137 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:17,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:17,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:17,137 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:17,137 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:17,137 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:17,137 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:17,137 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:17,138 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:17,138 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:17,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:17,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:17,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:17,138 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:17,138 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:17,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:17,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:17,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:17,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:17,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:17,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:17,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:17,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:17,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:17,140 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:17,140 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:17,140 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:17,140 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:17,140 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:17,141 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:17,141 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:17,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:17,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:17,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:17,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:17,141 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:17,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:17,141 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:17,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:17,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:17,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:17,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:17,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:17,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:17,142 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:17,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:17,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:17,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:17,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:17,144 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:17,144 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:17,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:17,144 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:17,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:17,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:17,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:17,144 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/d394dbf312894880b4aa7ed8f82ee058 2023-03-20 00:56:17,144 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:17,144 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:17,144 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:17,144 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:17,144 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:17,144 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:17,144 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:17,144 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:17,144 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:17,150 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/b2d9fb8e49174d54b0462b6179dd0c02 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/b2d9fb8e49174d54b0462b6179dd0c02 2023-03-20 00:56:17,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:17,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:17,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:17,160 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/b2d9fb8e49174d54b0462b6179dd0c02, entries=200, sequenceid=320, filesize=32.3 K 2023-03-20 00:56:17,161 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/07a9130d29d14684b29e95150f117ea6 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/07a9130d29d14684b29e95150f117ea6 2023-03-20 00:56:17,166 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/07a9130d29d14684b29e95150f117ea6, entries=150, sequenceid=320, filesize=11.9 K 2023-03-20 00:56:17,167 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/d394dbf312894880b4aa7ed8f82ee058 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/d394dbf312894880b4aa7ed8f82ee058 2023-03-20 00:56:17,171 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/d394dbf312894880b4aa7ed8f82ee058, entries=150, sequenceid=320, filesize=11.9 K 2023-03-20 00:56:17,171 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for f06a3c70774f5757abec4505bc3cdf91 in 964ms, sequenceid=320, compaction requested=false 2023-03-20 00:56:17,171 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:17,244 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:17,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:17,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:17,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:17,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:17,247 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:17,247 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:17,247 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:17,247 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:17,249 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:17,249 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:17,250 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:17,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:17,250 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:17,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:17,250 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:17,250 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:17,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:17,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:17,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:17,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:17,251 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:17,251 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:17,251 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:17,251 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:17,251 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:17,251 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:17,251 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:17,251 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:17,251 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:17,251 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. started... 2023-03-20 00:56:17,252 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f06a3c70774f5757abec4505bc3cdf91 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-20 00:56:17,252 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=A 2023-03-20 00:56:17,252 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:17,252 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=B 2023-03-20 00:56:17,252 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:17,252 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=C 2023-03-20 00:56:17,252 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:17,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:56:17,382 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. as already flushing 2023-03-20 00:56:17,428 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/55b8361a72104300b6359c586709433e as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/55b8361a72104300b6359c586709433e 2023-03-20 00:56:17,430 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:17,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:56226 deadline: 1679273837429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:17,438 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in f06a3c70774f5757abec4505bc3cdf91/A of f06a3c70774f5757abec4505bc3cdf91 into 55b8361a72104300b6359c586709433e(size=26.1 K), total size for store is 58.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-20 00:56:17,438 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:17,438 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91., storeName=f06a3c70774f5757abec4505bc3cdf91/A, priority=11, startTime=1679273776134; duration=1sec 2023-03-20 00:56:17,438 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:17,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:17,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:56226 deadline: 1679273837531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:17,671 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320e1c135b7c7d248e08275117b3c13fd88 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320e1c135b7c7d248e08275117b3c13fd88 2023-03-20 00:56:17,684 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=338, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/0533e923f79e4635ac33def656b7e6b7 2023-03-20 00:56:17,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:17,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:56226 deadline: 1679273837733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:18,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:18,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:56226 deadline: 1679273838038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:18,090 DEBUG [Thread-776] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bd79dff to 127.0.0.1:60547 2023-03-20 00:56:18,090 DEBUG [Thread-776] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:56:18,091 DEBUG [Thread-774] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50feca84 to 127.0.0.1:60547 2023-03-20 00:56:18,091 DEBUG [Thread-774] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:56:18,095 DEBUG [Thread-778] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52902392 to 127.0.0.1:60547 2023-03-20 00:56:18,095 DEBUG [Thread-780] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4e1da5ba to 127.0.0.1:60547 2023-03-20 00:56:18,095 DEBUG [Thread-780] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:56:18,095 DEBUG [Thread-778] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:56:18,104 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/0819cd7e1a6f4241a81f4290333c0d6e 2023-03-20 00:56:18,514 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/ccf02a2ae3414a438c6443512da16b80 2023-03-20 00:56:18,519 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/0533e923f79e4635ac33def656b7e6b7 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/0533e923f79e4635ac33def656b7e6b7 2023-03-20 00:56:18,524 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/0533e923f79e4635ac33def656b7e6b7, entries=150, sequenceid=338, filesize=25.4 K 2023-03-20 00:56:18,525 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/0819cd7e1a6f4241a81f4290333c0d6e as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/0819cd7e1a6f4241a81f4290333c0d6e 2023-03-20 00:56:18,529 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/0819cd7e1a6f4241a81f4290333c0d6e, entries=150, sequenceid=338, filesize=11.9 K 2023-03-20 00:56:18,529 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/ccf02a2ae3414a438c6443512da16b80 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/ccf02a2ae3414a438c6443512da16b80 2023-03-20 00:56:18,534 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/ccf02a2ae3414a438c6443512da16b80, entries=150, sequenceid=338, filesize=11.9 K 2023-03-20 00:56:18,534 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for f06a3c70774f5757abec4505bc3cdf91 in 1282ms, sequenceid=338, compaction requested=true 2023-03-20 00:56:18,535 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:18,535 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:18,535 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:18,535 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:18,535 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:18,535 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:18,535 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:18,536 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:18,536 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:18,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:18,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:18,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:18,536 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:18,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:18,536 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:18,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:18,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:18,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:18,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:18,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:18,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@317d53a4[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:18,538 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:18,538 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:18,538 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:18,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:18,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:18,539 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:18,539 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:18,539 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:18,539 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:18,539 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:18,539 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:18,539 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:18,539 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:18,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:18,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:18,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:18,540 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:18,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:56:18,540 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f06a3c70774f5757abec4505bc3cdf91 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-20 00:56:18,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:18,540 DEBUG [Thread-765] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b3311fc to 127.0.0.1:60547 2023-03-20 00:56:18,540 DEBUG [Thread-765] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:56:18,540 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=A 2023-03-20 00:56:18,541 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:18,541 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=B 2023-03-20 00:56:18,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:18,541 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:18,541 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=C 2023-03-20 00:56:18,541 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:18,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:18,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:18,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:18,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:18,543 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:18,543 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:18,543 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:18,543 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:18,543 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:18,544 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:18,544 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:18,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:18,544 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:18,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:18,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:18,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:18,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:18,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:18,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:18,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:18,545 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:18,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:18,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:18,546 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:18,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:18,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:18,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:18,548 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:18,548 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:18,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:18,548 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:18,548 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:18,548 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:18,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:18,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:18,548 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:18,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:18,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:18,548 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:18,548 DEBUG [Thread-773] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:18,548 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:18,548 DEBUG [Thread-773] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:18,548 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:18,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:18,548 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:18,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:18,648 DEBUG [Thread-773] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:18,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:18,955 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320350cb2d53688454bb7dbf45bbebacf13 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320350cb2d53688454bb7dbf45bbebacf13 2023-03-20 00:56:19,360 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=355, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/eb7c858030284ce088ceb398a23dfc1b 2023-03-20 00:56:19,773 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=355 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/9ff392941dc145eb802b91e81b88c6b5 2023-03-20 00:56:20,183 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=355 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/48a36fdac3fd43f6af8c5ab5c2727fbb 2023-03-20 00:56:20,188 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/eb7c858030284ce088ceb398a23dfc1b as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/eb7c858030284ce088ceb398a23dfc1b 2023-03-20 00:56:20,192 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/eb7c858030284ce088ceb398a23dfc1b, entries=150, sequenceid=355, filesize=25.4 K 2023-03-20 00:56:20,193 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/9ff392941dc145eb802b91e81b88c6b5 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/9ff392941dc145eb802b91e81b88c6b5 2023-03-20 00:56:20,197 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/9ff392941dc145eb802b91e81b88c6b5, entries=150, sequenceid=355, filesize=11.9 K 2023-03-20 00:56:20,198 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/48a36fdac3fd43f6af8c5ab5c2727fbb as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/48a36fdac3fd43f6af8c5ab5c2727fbb 2023-03-20 00:56:20,203 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/48a36fdac3fd43f6af8c5ab5c2727fbb, entries=150, sequenceid=355, filesize=11.9 K 2023-03-20 00:56:20,204 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for f06a3c70774f5757abec4505bc3cdf91 in 1664ms, sequenceid=355, compaction requested=true 2023-03-20 00:56:20,204 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:20,204 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:20,204 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 00:56:20,204 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 00:56:20,204 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:20,204 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:20,206 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111906 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 00:56:20,206 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49426 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 00:56:20,207 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): f06a3c70774f5757abec4505bc3cdf91/A is initiating minor compaction (all files) 2023-03-20 00:56:20,207 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): f06a3c70774f5757abec4505bc3cdf91/B is initiating minor compaction (all files) 2023-03-20 00:56:20,207 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f06a3c70774f5757abec4505bc3cdf91/A in TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:20,207 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f06a3c70774f5757abec4505bc3cdf91/B in TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:20,207 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/7d3f0e669cde4237b0f4d62c6ebbbd48, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/07a9130d29d14684b29e95150f117ea6, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/0819cd7e1a6f4241a81f4290333c0d6e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/9ff392941dc145eb802b91e81b88c6b5] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp, totalSize=48.3 K 2023-03-20 00:56:20,207 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/55b8361a72104300b6359c586709433e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/b2d9fb8e49174d54b0462b6179dd0c02, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/0533e923f79e4635ac33def656b7e6b7, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/eb7c858030284ce088ceb398a23dfc1b] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp, totalSize=109.3 K 2023-03-20 00:56:20,207 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 7d3f0e669cde4237b0f4d62c6ebbbd48, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1679273774400 2023-03-20 00:56:20,207 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 55b8361a72104300b6359c586709433e, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1679273774400 2023-03-20 00:56:20,208 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 07a9130d29d14684b29e95150f117ea6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1679273775558 2023-03-20 00:56:20,208 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting b2d9fb8e49174d54b0462b6179dd0c02, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1679273775558 2023-03-20 00:56:20,208 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 0819cd7e1a6f4241a81f4290333c0d6e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1679273776253 2023-03-20 00:56:20,208 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 0533e923f79e4635ac33def656b7e6b7, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1679273776253 2023-03-20 00:56:20,208 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 9ff392941dc145eb802b91e81b88c6b5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1679273777407 2023-03-20 00:56:20,208 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting eb7c858030284ce088ceb398a23dfc1b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1679273777407 2023-03-20 00:56:20,222 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): f06a3c70774f5757abec4505bc3cdf91#B#compaction#172 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:20,222 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): f06a3c70774f5757abec4505bc3cdf91#A#compaction#171 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:20,248 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/a0add558503f483f9fd44bdf1d0d622b as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/a0add558503f483f9fd44bdf1d0d622b 2023-03-20 00:56:20,254 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f06a3c70774f5757abec4505bc3cdf91/B of f06a3c70774f5757abec4505bc3cdf91 into a0add558503f483f9fd44bdf1d0d622b(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:20,254 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:20,254 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91., storeName=f06a3c70774f5757abec4505bc3cdf91/B, priority=12, startTime=1679273780204; duration=0sec 2023-03-20 00:56:20,254 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:20,254 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 00:56:20,255 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49426 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 00:56:20,255 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): f06a3c70774f5757abec4505bc3cdf91/C is initiating minor compaction (all files) 2023-03-20 00:56:20,255 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f06a3c70774f5757abec4505bc3cdf91/C in TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:20,255 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/13a9b92c731e401aaffc531a8cbab4e8, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/d394dbf312894880b4aa7ed8f82ee058, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/ccf02a2ae3414a438c6443512da16b80, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/48a36fdac3fd43f6af8c5ab5c2727fbb] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp, totalSize=48.3 K 2023-03-20 00:56:20,256 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 13a9b92c731e401aaffc531a8cbab4e8, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1679273774400 2023-03-20 00:56:20,256 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting d394dbf312894880b4aa7ed8f82ee058, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1679273775558 2023-03-20 00:56:20,256 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting ccf02a2ae3414a438c6443512da16b80, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1679273776253 2023-03-20 00:56:20,256 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 48a36fdac3fd43f6af8c5ab5c2727fbb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1679273777407 2023-03-20 00:56:20,273 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): f06a3c70774f5757abec4505bc3cdf91#C#compaction#173 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:20,684 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/8b24f7ce46dd4568824cce4b3eccb791 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/8b24f7ce46dd4568824cce4b3eccb791 2023-03-20 00:56:20,689 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f06a3c70774f5757abec4505bc3cdf91/C of f06a3c70774f5757abec4505bc3cdf91 into 8b24f7ce46dd4568824cce4b3eccb791(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:20,689 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:20,689 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91., storeName=f06a3c70774f5757abec4505bc3cdf91/C, priority=12, startTime=1679273780204; duration=0sec 2023-03-20 00:56:20,689 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:21,085 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/3db5ce5ed8ed47b49bd1003d3636c5e7 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/3db5ce5ed8ed47b49bd1003d3636c5e7 2023-03-20 00:56:21,090 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f06a3c70774f5757abec4505bc3cdf91/A of f06a3c70774f5757abec4505bc3cdf91 into 3db5ce5ed8ed47b49bd1003d3636c5e7(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:21,090 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:21,090 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91., storeName=f06a3c70774f5757abec4505bc3cdf91/A, priority=12, startTime=1679273780204; duration=0sec 2023-03-20 00:56:21,090 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:22,916 DEBUG [Thread-767] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x595355a0 to 127.0.0.1:60547 2023-03-20 00:56:22,916 DEBUG [Thread-767] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:56:22,931 DEBUG [Thread-771] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a84dba6 to 127.0.0.1:60547 2023-03-20 00:56:22,931 DEBUG [Thread-771] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:56:22,948 DEBUG [Thread-763] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x40eaa282 to 127.0.0.1:60547 2023-03-20 00:56:22,948 DEBUG [Thread-763] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:56:22,979 DEBUG [Thread-769] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7be739cc to 127.0.0.1:60547 2023-03-20 00:56:22,979 DEBUG [Thread-769] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:56:22,979 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-20 00:56:22,980 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(394): wrote 33 2023-03-20 00:56:22,980 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(394): wrote 140 2023-03-20 00:56:22,980 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(394): wrote 35 2023-03-20 00:56:22,980 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(394): wrote 36 2023-03-20 00:56:22,980 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(394): wrote 39 2023-03-20 00:56:22,980 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-20 00:56:22,980 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(398): read 4412 2023-03-20 00:56:22,981 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(398): read 4377 2023-03-20 00:56:22,981 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-20 00:56:22,981 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(402): scanned 1928 2023-03-20 00:56:22,981 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(403): verified 5781 rows 2023-03-20 00:56:22,981 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(402): scanned 1939 2023-03-20 00:56:22,981 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(403): verified 5817 rows 2023-03-20 00:56:22,981 INFO [Listener at localhost.localdomain/43361] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-03-20 00:56:22,981 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x54ecd46c to 127.0.0.1:60547 2023-03-20 00:56:22,981 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:56:22,982 INFO [Listener at localhost.localdomain/43361] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-20 00:56:22,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-03-20 00:56:22,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-20 00:56:22,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-20 00:56:22,987 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679273782987"}]},"ts":"1679273782987"} 2023-03-20 00:56:22,988 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-20 00:56:22,996 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-20 00:56:22,997 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f06a3c70774f5757abec4505bc3cdf91, UNASSIGN}] 2023-03-20 00:56:22,998 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f06a3c70774f5757abec4505bc3cdf91, UNASSIGN 2023-03-20 00:56:22,999 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=f06a3c70774f5757abec4505bc3cdf91, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:22,999 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273782999"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679273782999"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679273782999"}]},"ts":"1679273782999"} 2023-03-20 00:56:23,000 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199}] 2023-03-20 00:56:23,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-20 00:56:23,152 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:56:23,152 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing f06a3c70774f5757abec4505bc3cdf91, disabling compactions & flushes 2023-03-20 00:56:23,152 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:23,152 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:23,152 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. after waiting 0 ms 2023-03-20 00:56:23,152 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:23,152 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing f06a3c70774f5757abec4505bc3cdf91 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-20 00:56:23,152 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=A 2023-03-20 00:56:23,152 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:23,152 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=B 2023-03-20 00:56:23,152 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:23,152 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f06a3c70774f5757abec4505bc3cdf91, store=C 2023-03-20 00:56:23,152 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:23,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-20 00:56:23,566 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303208fea3ad081af4248a99b65983457ce64 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303208fea3ad081af4248a99b65983457ce64 2023-03-20 00:56:23,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-20 00:56:23,971 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=365, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/7b32bd5650e0492ca27875466e89f56b 2023-03-20 00:56:24,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-20 00:56:24,398 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=365 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/3ac1fd2a6bfa442396961207797ea75a 2023-03-20 00:56:24,836 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=365 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/0ab024d7a986458eab5056e12187bf35 2023-03-20 00:56:24,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/A/7b32bd5650e0492ca27875466e89f56b as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/7b32bd5650e0492ca27875466e89f56b 2023-03-20 00:56:24,848 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/7b32bd5650e0492ca27875466e89f56b, entries=150, sequenceid=365, filesize=25.4 K 2023-03-20 00:56:24,849 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/B/3ac1fd2a6bfa442396961207797ea75a as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/3ac1fd2a6bfa442396961207797ea75a 2023-03-20 00:56:24,855 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/3ac1fd2a6bfa442396961207797ea75a, entries=150, sequenceid=365, filesize=11.9 K 2023-03-20 00:56:24,857 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/.tmp/C/0ab024d7a986458eab5056e12187bf35 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/0ab024d7a986458eab5056e12187bf35 2023-03-20 00:56:24,864 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/0ab024d7a986458eab5056e12187bf35, entries=150, sequenceid=365, filesize=11.9 K 2023-03-20 00:56:24,865 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for f06a3c70774f5757abec4505bc3cdf91 in 1712ms, sequenceid=365, compaction requested=false 2023-03-20 00:56:24,885 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/2cb67ad6701d47b48529ceb7dbac393f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/e72b0c7214c24ee1b9163a2773c4c3b0, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/586b07b919e8477c99257d608daeea4f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/f2404f38f4ec4aed8f911bdd5cc4b5b3, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/246d67c0b06d420a9eb4f1e58997b850, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/71fa428390b14b0b95c14b139dc1a595, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/120a3bf4684e4ef4a290c70c40200a37, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/c5f782e1394b45238809df99261b49a4, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/8c669a1198e143f4a63435beb9880e6e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/5fb70bbd110a43e6b3347fd64598fb8c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/87fd9a4aa80b4af695bd530873ac9d13, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/7cd8c20e3c4d4da2b4057fab3eea61d3, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/726552578d4546bb83d305563cb29167, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/77cb88c105f44de18ff0e9e3bfad35b8, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/3b5295f32bbc46099a6c416648abeeb6, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/2ad4f74448a140cfa8a6c642e29beef6, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/6287161e224b48a0b02eded421d3792e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/ac2a045374914aec924493af4a18a8c9, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/e3d8297d602247559770b9464dcb912a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/55b8361a72104300b6359c586709433e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/ec6b308949314e829bdb37bc57bcbc9d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/b2d9fb8e49174d54b0462b6179dd0c02, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/0533e923f79e4635ac33def656b7e6b7, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/eb7c858030284ce088ceb398a23dfc1b] to archive 2023-03-20 00:56:24,886 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 00:56:24,887 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/2cb67ad6701d47b48529ceb7dbac393f to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/2cb67ad6701d47b48529ceb7dbac393f 2023-03-20 00:56:24,888 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/e72b0c7214c24ee1b9163a2773c4c3b0 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/e72b0c7214c24ee1b9163a2773c4c3b0 2023-03-20 00:56:24,889 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/586b07b919e8477c99257d608daeea4f to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/586b07b919e8477c99257d608daeea4f 2023-03-20 00:56:24,891 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/f2404f38f4ec4aed8f911bdd5cc4b5b3 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/f2404f38f4ec4aed8f911bdd5cc4b5b3 2023-03-20 00:56:24,892 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/246d67c0b06d420a9eb4f1e58997b850 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/246d67c0b06d420a9eb4f1e58997b850 2023-03-20 00:56:24,893 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/71fa428390b14b0b95c14b139dc1a595 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/71fa428390b14b0b95c14b139dc1a595 2023-03-20 00:56:24,894 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/120a3bf4684e4ef4a290c70c40200a37 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/120a3bf4684e4ef4a290c70c40200a37 2023-03-20 00:56:24,895 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/c5f782e1394b45238809df99261b49a4 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/c5f782e1394b45238809df99261b49a4 2023-03-20 00:56:24,896 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/8c669a1198e143f4a63435beb9880e6e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/8c669a1198e143f4a63435beb9880e6e 2023-03-20 00:56:24,897 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/5fb70bbd110a43e6b3347fd64598fb8c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/5fb70bbd110a43e6b3347fd64598fb8c 2023-03-20 00:56:24,897 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/87fd9a4aa80b4af695bd530873ac9d13 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/87fd9a4aa80b4af695bd530873ac9d13 2023-03-20 00:56:24,899 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/7cd8c20e3c4d4da2b4057fab3eea61d3 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/7cd8c20e3c4d4da2b4057fab3eea61d3 2023-03-20 00:56:24,900 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/726552578d4546bb83d305563cb29167 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/726552578d4546bb83d305563cb29167 2023-03-20 00:56:24,901 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/77cb88c105f44de18ff0e9e3bfad35b8 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/77cb88c105f44de18ff0e9e3bfad35b8 2023-03-20 00:56:24,902 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/3b5295f32bbc46099a6c416648abeeb6 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/3b5295f32bbc46099a6c416648abeeb6 2023-03-20 00:56:24,903 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/2ad4f74448a140cfa8a6c642e29beef6 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/2ad4f74448a140cfa8a6c642e29beef6 2023-03-20 00:56:24,904 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/6287161e224b48a0b02eded421d3792e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/6287161e224b48a0b02eded421d3792e 2023-03-20 00:56:24,905 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/ac2a045374914aec924493af4a18a8c9 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/ac2a045374914aec924493af4a18a8c9 2023-03-20 00:56:24,906 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/e3d8297d602247559770b9464dcb912a to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/e3d8297d602247559770b9464dcb912a 2023-03-20 00:56:24,907 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/55b8361a72104300b6359c586709433e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/55b8361a72104300b6359c586709433e 2023-03-20 00:56:24,908 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/ec6b308949314e829bdb37bc57bcbc9d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/ec6b308949314e829bdb37bc57bcbc9d 2023-03-20 00:56:24,909 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/b2d9fb8e49174d54b0462b6179dd0c02 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/b2d9fb8e49174d54b0462b6179dd0c02 2023-03-20 00:56:24,910 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/0533e923f79e4635ac33def656b7e6b7 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/0533e923f79e4635ac33def656b7e6b7 2023-03-20 00:56:24,911 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/eb7c858030284ce088ceb398a23dfc1b to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/eb7c858030284ce088ceb398a23dfc1b 2023-03-20 00:56:24,914 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/dae19c07a69e4966a217cfd1c4a37fd4, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/559f78dcdd6f4af493f2c79f01cd7997, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/e5b3e937950b4b9e81722ba4600d924e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/f71ebd84f6d4433985b19c884b2f2148, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/8c2d5dca21be45cdaec09dff4412956c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/2d80ed2d849d4b7991ef129825b9647d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/8d01404fb75440938a95659791116f4f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/ddcf88331ff14d68a63d62ef4286644b, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/8ed0bcc71f1f410bbaa8d3e11188501d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/c2ade211bd694abebacaedeac504a3ec, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/8a09574f74f64107a70e91a1505512b6, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/940e7d64c004464882f4ab983c21f277, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/377a137bb2554784a942f581826b358b, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/9d39977c9f264cc08028f3c8e8e1e30c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/d278b1a3082a4d51ac5db87a21adfc1f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/dd53309d29e64eb39c9f6be1333f5d43, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/c56768c7101d47d7985482cceef96a97, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/48c80e1306c246549d1eade290fa18f0, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/518a13b594d94818b116a46bb24feff2, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/7d3f0e669cde4237b0f4d62c6ebbbd48, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/5f04d41e5cd74add9a427a38dd0eca13, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/07a9130d29d14684b29e95150f117ea6, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/0819cd7e1a6f4241a81f4290333c0d6e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/9ff392941dc145eb802b91e81b88c6b5] to archive 2023-03-20 00:56:24,915 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 00:56:24,917 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/dae19c07a69e4966a217cfd1c4a37fd4 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/dae19c07a69e4966a217cfd1c4a37fd4 2023-03-20 00:56:24,918 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/559f78dcdd6f4af493f2c79f01cd7997 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/559f78dcdd6f4af493f2c79f01cd7997 2023-03-20 00:56:24,919 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/e5b3e937950b4b9e81722ba4600d924e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/e5b3e937950b4b9e81722ba4600d924e 2023-03-20 00:56:24,921 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/f71ebd84f6d4433985b19c884b2f2148 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/f71ebd84f6d4433985b19c884b2f2148 2023-03-20 00:56:24,922 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/8c2d5dca21be45cdaec09dff4412956c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/8c2d5dca21be45cdaec09dff4412956c 2023-03-20 00:56:24,923 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/2d80ed2d849d4b7991ef129825b9647d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/2d80ed2d849d4b7991ef129825b9647d 2023-03-20 00:56:24,924 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/8d01404fb75440938a95659791116f4f to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/8d01404fb75440938a95659791116f4f 2023-03-20 00:56:24,925 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/ddcf88331ff14d68a63d62ef4286644b to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/ddcf88331ff14d68a63d62ef4286644b 2023-03-20 00:56:24,926 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/8ed0bcc71f1f410bbaa8d3e11188501d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/8ed0bcc71f1f410bbaa8d3e11188501d 2023-03-20 00:56:24,927 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/c2ade211bd694abebacaedeac504a3ec to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/c2ade211bd694abebacaedeac504a3ec 2023-03-20 00:56:24,928 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/8a09574f74f64107a70e91a1505512b6 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/8a09574f74f64107a70e91a1505512b6 2023-03-20 00:56:24,929 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/940e7d64c004464882f4ab983c21f277 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/940e7d64c004464882f4ab983c21f277 2023-03-20 00:56:24,930 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/377a137bb2554784a942f581826b358b to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/377a137bb2554784a942f581826b358b 2023-03-20 00:56:24,931 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/9d39977c9f264cc08028f3c8e8e1e30c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/9d39977c9f264cc08028f3c8e8e1e30c 2023-03-20 00:56:24,932 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/d278b1a3082a4d51ac5db87a21adfc1f to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/d278b1a3082a4d51ac5db87a21adfc1f 2023-03-20 00:56:24,933 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/dd53309d29e64eb39c9f6be1333f5d43 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/dd53309d29e64eb39c9f6be1333f5d43 2023-03-20 00:56:24,934 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/c56768c7101d47d7985482cceef96a97 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/c56768c7101d47d7985482cceef96a97 2023-03-20 00:56:24,936 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/48c80e1306c246549d1eade290fa18f0 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/48c80e1306c246549d1eade290fa18f0 2023-03-20 00:56:24,937 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/518a13b594d94818b116a46bb24feff2 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/518a13b594d94818b116a46bb24feff2 2023-03-20 00:56:24,939 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/7d3f0e669cde4237b0f4d62c6ebbbd48 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/7d3f0e669cde4237b0f4d62c6ebbbd48 2023-03-20 00:56:24,940 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/5f04d41e5cd74add9a427a38dd0eca13 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/5f04d41e5cd74add9a427a38dd0eca13 2023-03-20 00:56:24,941 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/07a9130d29d14684b29e95150f117ea6 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/07a9130d29d14684b29e95150f117ea6 2023-03-20 00:56:24,941 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/0819cd7e1a6f4241a81f4290333c0d6e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/0819cd7e1a6f4241a81f4290333c0d6e 2023-03-20 00:56:24,942 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/9ff392941dc145eb802b91e81b88c6b5 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/9ff392941dc145eb802b91e81b88c6b5 2023-03-20 00:56:24,945 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/f7ec1a873fb344e2ad1b78fd0b9cade4, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/6387bb4c161f4446bb57ac2696d55f02, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/2986d963dbfa485c94c9afa6e983e299, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/e0d1ac94575f46518634ba724284fb27, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/33bdd917dca6490f9ba0bd0cd33aaf46, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/95de81ae673b4ee987c1af7925ef1d08, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/00d0b06fb8f2479794cbfa227d36e210, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/e9d4c3cd33af444f87b72e8859f6c10a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/8423f8f8a7a54f248a6d705ce555f65d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/063f2a5c739047d2bb64b92e3b89aca2, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/54a09d3700614249b1533c622787cfcb, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/c1d17a093eeb440e8a99f00af145f8bc, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/24741bbcd5e2464a8f9705c08137ac65, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/382486d354594cb5b71fd0cb7b97da9a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/9b28273df1a84012b8c698a8ae374a45, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/139b2e9894ba476ca375bb9c8a6bd448, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/ee10cac1ba0746a08c019041f2a203f2, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/57f537bf126f47d5aa1c3c827f32f700, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/d6dcb9ef94184e16b569e72b1bc2ea2a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/13a9b92c731e401aaffc531a8cbab4e8, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/218b9184bf8349c7a2a79ddfac4e7e1e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/d394dbf312894880b4aa7ed8f82ee058, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/ccf02a2ae3414a438c6443512da16b80, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/48a36fdac3fd43f6af8c5ab5c2727fbb] to archive 2023-03-20 00:56:24,945 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 00:56:24,947 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/f7ec1a873fb344e2ad1b78fd0b9cade4 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/f7ec1a873fb344e2ad1b78fd0b9cade4 2023-03-20 00:56:24,948 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/6387bb4c161f4446bb57ac2696d55f02 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/6387bb4c161f4446bb57ac2696d55f02 2023-03-20 00:56:24,949 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/2986d963dbfa485c94c9afa6e983e299 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/2986d963dbfa485c94c9afa6e983e299 2023-03-20 00:56:24,950 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/e0d1ac94575f46518634ba724284fb27 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/e0d1ac94575f46518634ba724284fb27 2023-03-20 00:56:24,951 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/33bdd917dca6490f9ba0bd0cd33aaf46 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/33bdd917dca6490f9ba0bd0cd33aaf46 2023-03-20 00:56:24,952 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/95de81ae673b4ee987c1af7925ef1d08 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/95de81ae673b4ee987c1af7925ef1d08 2023-03-20 00:56:24,953 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/00d0b06fb8f2479794cbfa227d36e210 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/00d0b06fb8f2479794cbfa227d36e210 2023-03-20 00:56:24,954 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/e9d4c3cd33af444f87b72e8859f6c10a to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/e9d4c3cd33af444f87b72e8859f6c10a 2023-03-20 00:56:24,955 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/8423f8f8a7a54f248a6d705ce555f65d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/8423f8f8a7a54f248a6d705ce555f65d 2023-03-20 00:56:24,956 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/063f2a5c739047d2bb64b92e3b89aca2 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/063f2a5c739047d2bb64b92e3b89aca2 2023-03-20 00:56:24,957 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/54a09d3700614249b1533c622787cfcb to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/54a09d3700614249b1533c622787cfcb 2023-03-20 00:56:24,958 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/c1d17a093eeb440e8a99f00af145f8bc to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/c1d17a093eeb440e8a99f00af145f8bc 2023-03-20 00:56:24,959 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/24741bbcd5e2464a8f9705c08137ac65 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/24741bbcd5e2464a8f9705c08137ac65 2023-03-20 00:56:24,960 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/382486d354594cb5b71fd0cb7b97da9a to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/382486d354594cb5b71fd0cb7b97da9a 2023-03-20 00:56:24,961 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/9b28273df1a84012b8c698a8ae374a45 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/9b28273df1a84012b8c698a8ae374a45 2023-03-20 00:56:24,962 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/139b2e9894ba476ca375bb9c8a6bd448 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/139b2e9894ba476ca375bb9c8a6bd448 2023-03-20 00:56:24,963 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/ee10cac1ba0746a08c019041f2a203f2 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/ee10cac1ba0746a08c019041f2a203f2 2023-03-20 00:56:24,964 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/57f537bf126f47d5aa1c3c827f32f700 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/57f537bf126f47d5aa1c3c827f32f700 2023-03-20 00:56:24,965 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/d6dcb9ef94184e16b569e72b1bc2ea2a to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/d6dcb9ef94184e16b569e72b1bc2ea2a 2023-03-20 00:56:24,966 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/13a9b92c731e401aaffc531a8cbab4e8 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/13a9b92c731e401aaffc531a8cbab4e8 2023-03-20 00:56:24,967 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/218b9184bf8349c7a2a79ddfac4e7e1e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/218b9184bf8349c7a2a79ddfac4e7e1e 2023-03-20 00:56:24,968 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/d394dbf312894880b4aa7ed8f82ee058 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/d394dbf312894880b4aa7ed8f82ee058 2023-03-20 00:56:24,969 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/ccf02a2ae3414a438c6443512da16b80 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/ccf02a2ae3414a438c6443512da16b80 2023-03-20 00:56:24,970 DEBUG [StoreCloser-TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/48a36fdac3fd43f6af8c5ab5c2727fbb to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/48a36fdac3fd43f6af8c5ab5c2727fbb 2023-03-20 00:56:24,973 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/recovered.edits/368.seqid, newMaxSeqId=368, maxSeqId=4 2023-03-20 00:56:24,975 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91. 2023-03-20 00:56:24,975 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for f06a3c70774f5757abec4505bc3cdf91: 2023-03-20 00:56:24,976 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:56:24,977 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=f06a3c70774f5757abec4505bc3cdf91, regionState=CLOSED 2023-03-20 00:56:24,977 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273784977"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679273784977"}]},"ts":"1679273784977"} 2023-03-20 00:56:24,979 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-03-20 00:56:24,979 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure f06a3c70774f5757abec4505bc3cdf91, server=jenkins-hbase20.apache.org,34553,1679273725199 in 1.9780 sec 2023-03-20 00:56:24,980 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-03-20 00:56:24,980 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=f06a3c70774f5757abec4505bc3cdf91, UNASSIGN in 1.9820 sec 2023-03-20 00:56:24,981 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679273784981"}]},"ts":"1679273784981"} 2023-03-20 00:56:24,982 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-20 00:56:24,983 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-20 00:56:24,984 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.0000 sec 2023-03-20 00:56:25,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-20 00:56:25,092 INFO [Listener at localhost.localdomain/43361] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-03-20 00:56:25,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-03-20 00:56:25,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:56:25,094 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:56:25,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-20 00:56:25,095 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:56:25,099 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:56:25,101 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A, FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B, FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C, FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/recovered.edits] 2023-03-20 00:56:25,104 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/3db5ce5ed8ed47b49bd1003d3636c5e7 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/3db5ce5ed8ed47b49bd1003d3636c5e7 2023-03-20 00:56:25,105 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/7b32bd5650e0492ca27875466e89f56b to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/A/7b32bd5650e0492ca27875466e89f56b 2023-03-20 00:56:25,107 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/3ac1fd2a6bfa442396961207797ea75a to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/3ac1fd2a6bfa442396961207797ea75a 2023-03-20 00:56:25,109 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/a0add558503f483f9fd44bdf1d0d622b to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/B/a0add558503f483f9fd44bdf1d0d622b 2023-03-20 00:56:25,110 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/0ab024d7a986458eab5056e12187bf35 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/0ab024d7a986458eab5056e12187bf35 2023-03-20 00:56:25,111 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/8b24f7ce46dd4568824cce4b3eccb791 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/C/8b24f7ce46dd4568824cce4b3eccb791 2023-03-20 00:56:25,113 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/recovered.edits/368.seqid to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91/recovered.edits/368.seqid 2023-03-20 00:56:25,114 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/f06a3c70774f5757abec4505bc3cdf91 2023-03-20 00:56:25,114 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-20 00:56:25,114 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-20 00:56:25,115 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-20 00:56:25,118 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303200303eadf496e41dabfeee8dd5191fa33 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303200303eadf496e41dabfeee8dd5191fa33 2023-03-20 00:56:25,119 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032005924189bedd43bab1815fb6fd86c036 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032005924189bedd43bab1815fb6fd86c036 2023-03-20 00:56:25,120 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303201e9eda4131b040ec87ad1c4a151855aa to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303201e9eda4131b040ec87ad1c4a151855aa 2023-03-20 00:56:25,121 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032020b0d4dc308e46eb94d952d3ce01beac to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032020b0d4dc308e46eb94d952d3ce01beac 2023-03-20 00:56:25,123 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032027967a04bd1b4f21aa4973def16a3a80 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032027967a04bd1b4f21aa4973def16a3a80 2023-03-20 00:56:25,124 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320350cb2d53688454bb7dbf45bbebacf13 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320350cb2d53688454bb7dbf45bbebacf13 2023-03-20 00:56:25,125 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032040333ac79d7349f88df816633dfde872 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032040333ac79d7349f88df816633dfde872 2023-03-20 00:56:25,126 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303204f636a72bff44b6e93f52f2bce61ad2c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303204f636a72bff44b6e93f52f2bce61ad2c 2023-03-20 00:56:25,128 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303206c87dae1870b4fe9ad15c420e4443449 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303206c87dae1870b4fe9ad15c420e4443449 2023-03-20 00:56:25,129 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303207e26244bbaa34ded99c9ca8f47acac71 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303207e26244bbaa34ded99c9ca8f47acac71 2023-03-20 00:56:25,130 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303208fb921e4084f4a089418863c83735500 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303208fb921e4084f4a089418863c83735500 2023-03-20 00:56:25,132 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303208fea3ad081af4248a99b65983457ce64 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303208fea3ad081af4248a99b65983457ce64 2023-03-20 00:56:25,133 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303209384435f1c0a4aeda086bcc4949fd05d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303209384435f1c0a4aeda086bcc4949fd05d 2023-03-20 00:56:25,134 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320b23f54e965b3474e8b656b1299b41a53 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320b23f54e965b3474e8b656b1299b41a53 2023-03-20 00:56:25,135 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320c68fa760a5b74628b673406711cf0c88 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320c68fa760a5b74628b673406711cf0c88 2023-03-20 00:56:25,136 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320c708d36f37be4281be705cd7bde0177d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320c708d36f37be4281be705cd7bde0177d 2023-03-20 00:56:25,137 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320c8e6e452dcdf4bb59d66b9bc0f327adf to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320c8e6e452dcdf4bb59d66b9bc0f327adf 2023-03-20 00:56:25,139 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320c908dfa62b0e42489d460d116431c6dd to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320c908dfa62b0e42489d460d116431c6dd 2023-03-20 00:56:25,140 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320e1c135b7c7d248e08275117b3c13fd88 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320e1c135b7c7d248e08275117b3c13fd88 2023-03-20 00:56:25,141 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320ee4e18bfd58142f9abb61e703c2a010e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320ee4e18bfd58142f9abb61e703c2a010e 2023-03-20 00:56:25,142 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320f3864e092284432696023c5ff79d6d13 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320f3864e092284432696023c5ff79d6d13 2023-03-20 00:56:25,142 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-20 00:56:25,144 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:56:25,145 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-20 00:56:25,147 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-20 00:56:25,148 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:56:25,148 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-20 00:56:25,148 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679273785148"}]},"ts":"9223372036854775807"} 2023-03-20 00:56:25,150 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-20 00:56:25,150 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => f06a3c70774f5757abec4505bc3cdf91, NAME => 'TestAcidGuarantees,,1679273754892.f06a3c70774f5757abec4505bc3cdf91.', STARTKEY => '', ENDKEY => ''}] 2023-03-20 00:56:25,150 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-20 00:56:25,150 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679273785150"}]},"ts":"9223372036854775807"} 2023-03-20 00:56:25,152 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-20 00:56:25,153 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:56:25,154 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 61 msec 2023-03-20 00:56:25,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-20 00:56:25,196 INFO [Listener at localhost.localdomain/43361] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-03-20 00:56:25,210 INFO [Listener at localhost.localdomain/43361] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=230 (was 234), OpenFileDescriptor=545 (was 545), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=392 (was 334) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=4855 (was 4638) - AvailableMemoryMB LEAK? - 2023-03-20 00:56:25,224 INFO [Listener at localhost.localdomain/43361] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=230, OpenFileDescriptor=545, MaxFileDescriptor=60000, SystemLoadAverage=392, ProcessCount=168, AvailableMemoryMB=4855 2023-03-20 00:56:25,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-20 00:56:25,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-20 00:56:25,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-20 00:56:25,229 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-20 00:56:25,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-03-20 00:56:25,230 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-20 00:56:25,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-20 00:56:25,231 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:25,231 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a empty. 2023-03-20 00:56:25,232 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:25,232 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-20 00:56:25,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-20 00:56:25,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-20 00:56:25,641 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-20 00:56:25,643 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 51e645e88856df580ad0de225d47b16a, NAME => 'TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp 2023-03-20 00:56:25,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-20 00:56:26,060 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 00:56:26,060 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 51e645e88856df580ad0de225d47b16a, disabling compactions & flushes 2023-03-20 00:56:26,060 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:26,060 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:26,060 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. after waiting 0 ms 2023-03-20 00:56:26,060 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:26,060 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:26,060 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:26,063 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-20 00:56:26,063 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273786063"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679273786063"}]},"ts":"1679273786063"} 2023-03-20 00:56:26,064 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-20 00:56:26,065 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-20 00:56:26,065 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679273786065"}]},"ts":"1679273786065"} 2023-03-20 00:56:26,066 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-20 00:56:26,075 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=51e645e88856df580ad0de225d47b16a, ASSIGN}] 2023-03-20 00:56:26,076 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=51e645e88856df580ad0de225d47b16a, ASSIGN 2023-03-20 00:56:26,077 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=51e645e88856df580ad0de225d47b16a, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34553,1679273725199; forceNewPlan=false, retain=false 2023-03-20 00:56:26,228 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=51e645e88856df580ad0de225d47b16a, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:26,228 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273786228"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679273786228"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679273786228"}]},"ts":"1679273786228"} 2023-03-20 00:56:26,230 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199}] 2023-03-20 00:56:26,297 DEBUG [regionserver/jenkins-hbase20:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-20 00:56:26,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-20 00:56:26,392 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:26,392 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 51e645e88856df580ad0de225d47b16a, NAME => 'TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.', STARTKEY => '', ENDKEY => ''} 2023-03-20 00:56:26,392 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:26,393 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 00:56:26,393 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:26,393 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:26,400 INFO [StoreOpener-51e645e88856df580ad0de225d47b16a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:26,409 DEBUG [StoreOpener-51e645e88856df580ad0de225d47b16a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A 2023-03-20 00:56:26,409 DEBUG [StoreOpener-51e645e88856df580ad0de225d47b16a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A 2023-03-20 00:56:26,409 INFO [StoreOpener-51e645e88856df580ad0de225d47b16a-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 00:56:26,409 INFO [StoreOpener-51e645e88856df580ad0de225d47b16a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 51e645e88856df580ad0de225d47b16a columnFamilyName A 2023-03-20 00:56:26,410 INFO [StoreOpener-51e645e88856df580ad0de225d47b16a-1] regionserver.HStore(310): Store=51e645e88856df580ad0de225d47b16a/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:56:26,411 INFO [StoreOpener-51e645e88856df580ad0de225d47b16a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:26,413 DEBUG [StoreOpener-51e645e88856df580ad0de225d47b16a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B 2023-03-20 00:56:26,413 DEBUG [StoreOpener-51e645e88856df580ad0de225d47b16a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B 2023-03-20 00:56:26,413 INFO [StoreOpener-51e645e88856df580ad0de225d47b16a-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 00:56:26,414 INFO [StoreOpener-51e645e88856df580ad0de225d47b16a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 51e645e88856df580ad0de225d47b16a columnFamilyName B 2023-03-20 00:56:26,414 INFO [StoreOpener-51e645e88856df580ad0de225d47b16a-1] regionserver.HStore(310): Store=51e645e88856df580ad0de225d47b16a/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:56:26,414 INFO [StoreOpener-51e645e88856df580ad0de225d47b16a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:26,416 DEBUG [StoreOpener-51e645e88856df580ad0de225d47b16a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C 2023-03-20 00:56:26,416 DEBUG [StoreOpener-51e645e88856df580ad0de225d47b16a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C 2023-03-20 00:56:26,416 INFO [StoreOpener-51e645e88856df580ad0de225d47b16a-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 00:56:26,416 INFO [StoreOpener-51e645e88856df580ad0de225d47b16a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 51e645e88856df580ad0de225d47b16a columnFamilyName C 2023-03-20 00:56:26,417 INFO [StoreOpener-51e645e88856df580ad0de225d47b16a-1] regionserver.HStore(310): Store=51e645e88856df580ad0de225d47b16a/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:56:26,417 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:26,418 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:26,418 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:26,420 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-20 00:56:26,421 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:26,424 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-20 00:56:26,425 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 51e645e88856df580ad0de225d47b16a; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71754800, jitterRate=0.0692298412322998}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-20 00:56:26,425 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:26,425 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a., pid=30, masterSystemTime=1679273786381 2023-03-20 00:56:26,427 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:26,427 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:26,427 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=51e645e88856df580ad0de225d47b16a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:26,427 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273786427"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679273786427"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679273786427"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679273786427"}]},"ts":"1679273786427"} 2023-03-20 00:56:26,430 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-03-20 00:56:26,430 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 in 198 msec 2023-03-20 00:56:26,432 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-03-20 00:56:26,432 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=51e645e88856df580ad0de225d47b16a, ASSIGN in 355 msec 2023-03-20 00:56:26,432 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-20 00:56:26,433 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679273786433"}]},"ts":"1679273786433"} 2023-03-20 00:56:26,434 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-20 00:56:26,436 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-20 00:56:26,439 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2100 sec 2023-03-20 00:56:27,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-20 00:56:27,337 INFO [Listener at localhost.localdomain/43361] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-03-20 00:56:27,339 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x7117e4df to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:56:27,345 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@136e067d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:56:27,346 DEBUG [Listener at localhost.localdomain/43361] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:56:27,348 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48342, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:56:27,349 DEBUG [Listener at localhost.localdomain/43361] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-20 00:56:27,351 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:46290, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-20 00:56:27,352 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x5c466b6c to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:56:27,356 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12394d27, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:56:27,356 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x46d74611 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:56:27,359 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29a548ac, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:56:27,360 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x3f837278 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:56:27,365 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43184c57, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:56:27,365 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x5a8a575d to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:56:27,369 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e3efec6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:56:27,370 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x6347f2e6 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:56:27,373 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6b7f716c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:56:27,374 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x6ade356f to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:56:27,378 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7abacd49, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:56:27,378 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x4cbc74e7 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:56:27,382 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@57d25552, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:56:27,383 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x3c5d1710 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:56:27,405 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13695730, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:56:27,406 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x5d22a55e to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:56:27,414 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@993cc7d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:56:27,414 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x2179ead9 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:56:27,422 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69b9b3fd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:56:27,428 DEBUG [hconnection-0x44fbd99f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:56:27,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:27,432 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48356, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:56:27,433 DEBUG [hconnection-0x2e58a768-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:56:27,434 DEBUG [hconnection-0x59fa15af-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:56:27,435 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48380, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:56:27,438 DEBUG [hconnection-0x1f4f9ee8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:56:27,440 DEBUG [hconnection-0x526b3379-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:56:27,439 DEBUG [hconnection-0x49fe1683-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:56:27,440 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48368, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:56:27,440 DEBUG [hconnection-0x792dcfe0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:56:27,440 DEBUG [hconnection-0x69ab9f6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:56:27,441 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48408, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:56:27,440 DEBUG [hconnection-0x6416e408-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:56:27,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:27,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:27,440 DEBUG [hconnection-0x4aae601a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:56:27,442 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:27,442 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:27,442 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:27,442 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:27,443 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48428, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:56:27,443 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:27,443 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:27,444 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:27,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:27,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:27,445 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:27,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:27,447 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:27,447 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:27,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:27,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:27,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:27,448 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:27,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:27,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:27,449 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:27,449 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:27,449 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:27,450 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48396, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:56:27,450 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:27,450 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:27,450 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:27,450 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:27,451 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48422, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:56:27,454 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48388, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:56:27,454 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48410, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:56:27,455 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48442, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:56:27,459 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51e645e88856df580ad0de225d47b16a 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-03-20 00:56:27,467 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=A 2023-03-20 00:56:27,468 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:27,473 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=B 2023-03-20 00:56:27,473 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:27,474 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=C 2023-03-20 00:56:27,474 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:27,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:27,475 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:27,495 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679273787482, max timestamp=1679273787493Num uniques 859; 2023-03-20 00:56:27,495 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 00:56:27,495 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679273787482, max timestamp=1679273787491Num uniques 859; 2023-03-20 00:56:27,496 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 00:56:27,496 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=200, cellSize=9160, totalHeapSize=21040, min timestamp=1679273787482, max timestamp=1679273787493Num uniques 859; 2023-03-20 00:56:27,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9160, suffix heap size=117064,new segment heap size=21040 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=200 2023-03-20 00:56:27,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679273787504, max timestamp=1679273787507Num uniques 1068; 2023-03-20 00:56:27,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-20 00:56:27,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12267, totalHeapSize=28112, min timestamp=1679273787504, max timestamp=1679273787508Num uniques 1068; 2023-03-20 00:56:27,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48907, new segment data size=12267, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-20 00:56:27,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12264, totalHeapSize=28112, min timestamp=1679273787504, max timestamp=1679273787508Num uniques 1059; 2023-03-20 00:56:27,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48499, new segment data size=12264, suffix heap size=138104,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1059, new segment cells count=268 2023-03-20 00:56:27,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:27,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48410 deadline: 1679273847519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:27,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:27,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48408 deadline: 1679273847520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:27,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679273787515, max timestamp=1679273787520Num uniques 1127; 2023-03-20 00:56:27,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-03-20 00:56:27,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10382, totalHeapSize=23848, min timestamp=1679273787515, max timestamp=1679273787521Num uniques 1127; 2023-03-20 00:56:27,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=51602, new segment data size=10382, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-03-20 00:56:27,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10391, totalHeapSize=23848, min timestamp=1679273787515, max timestamp=1679273787521Num uniques 1127; 2023-03-20 00:56:27,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=51611, new segment data size=10391, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-03-20 00:56:27,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679273787526, max timestamp=1679273787541Num uniques 1086; 2023-03-20 00:56:27,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679273787526, max timestamp=1679273787541Num uniques 1086; 2023-03-20 00:56:27,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-20 00:56:27,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-20 00:56:27,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679273787526, max timestamp=1679273787541Num uniques 1086; 2023-03-20 00:56:27,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-20 00:56:27,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679273787552, max timestamp=1679273787556Num uniques 1045; 2023-03-20 00:56:27,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-20 00:56:27,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679273787552, max timestamp=1679273787556Num uniques 1045; 2023-03-20 00:56:27,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-20 00:56:27,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679273787552, max timestamp=1679273787556Num uniques 1045; 2023-03-20 00:56:27,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-20 00:56:27,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679273787572, max timestamp=1679273787580Num uniques 1104; 2023-03-20 00:56:27,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679273787572, max timestamp=1679273787580Num uniques 1104; 2023-03-20 00:56:27,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-20 00:56:27,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-20 00:56:27,587 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679273787572, max timestamp=1679273787580Num uniques 1104; 2023-03-20 00:56:27,587 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-20 00:56:27,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679273787596, max timestamp=1679273787603Num uniques 1013; 2023-03-20 00:56:27,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679273787596, max timestamp=1679273787603Num uniques 1013; 2023-03-20 00:56:27,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679273787596, max timestamp=1679273787603Num uniques 1013; 2023-03-20 00:56:27,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-03-20 00:56:27,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-03-20 00:56:27,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-03-20 00:56:27,641 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:27,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:27,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48422 deadline: 1679273847639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:27,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48368 deadline: 1679273847639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:27,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7863, totalHeapSize=18128, min timestamp=1679273787616, max timestamp=1679273787640Num uniques 1072; 2023-03-20 00:56:27,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49083, new segment data size=7863, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-03-20 00:56:27,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7865, totalHeapSize=18128, min timestamp=1679273787616, max timestamp=1679273787640Num uniques 1072; 2023-03-20 00:56:27,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49085, new segment data size=7865, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-03-20 00:56:27,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=155, cellSize=7095, totalHeapSize=16360, min timestamp=1679273787616, max timestamp=1679273787640Num uniques 1072; 2023-03-20 00:56:27,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49087, new segment data size=7095, suffix heap size=139456,new segment heap size=16360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=155 2023-03-20 00:56:27,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679273787652, max timestamp=1679273787655Num uniques 1014; 2023-03-20 00:56:27,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46444, new segment data size=8286, suffix heap size=133424,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1014, new segment cells count=181 2023-03-20 00:56:27,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679273787649, max timestamp=1679273787655Num uniques 1031; 2023-03-20 00:56:27,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-03-20 00:56:27,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679273787652, max timestamp=1679273787655Num uniques 1031; 2023-03-20 00:56:27,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-03-20 00:56:27,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10983, totalHeapSize=25200, min timestamp=1679273787662, max timestamp=1679273787668Num uniques 1040; 2023-03-20 00:56:27,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10980, totalHeapSize=25200, min timestamp=1679273787662, max timestamp=1679273787668Num uniques 1090; 2023-03-20 00:56:27,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10986, totalHeapSize=25200, min timestamp=1679273787662, max timestamp=1679273787668Num uniques 1040; 2023-03-20 00:56:27,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49910, new segment data size=10980, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-03-20 00:56:27,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47623, new segment data size=10983, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-03-20 00:56:27,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47626, new segment data size=10986, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-03-20 00:56:27,704 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1679273787694, max timestamp=1679273787702Num uniques 1099; 2023-03-20 00:56:27,704 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-03-20 00:56:27,704 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1679273787694, max timestamp=1679273787702Num uniques 1099; 2023-03-20 00:56:27,704 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-03-20 00:56:27,705 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1679273787694, max timestamp=1679273787702Num uniques 1099; 2023-03-20 00:56:27,705 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-03-20 00:56:27,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:27,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1679273787711, max timestamp=1679273787717Num uniques 1108; 2023-03-20 00:56:27,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48428 deadline: 1679273847717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:27,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-03-20 00:56:27,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1679273787711, max timestamp=1679273787717Num uniques 1108; 2023-03-20 00:56:27,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1679273787711, max timestamp=1679273787717Num uniques 1108; 2023-03-20 00:56:27,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-03-20 00:56:27,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-03-20 00:56:27,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679273787735, max timestamp=1679273787739Num uniques 1067; 2023-03-20 00:56:27,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-03-20 00:56:27,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679273787735, max timestamp=1679273787739Num uniques 1067; 2023-03-20 00:56:27,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-03-20 00:56:27,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679273787735, max timestamp=1679273787739Num uniques 1067; 2023-03-20 00:56:27,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-03-20 00:56:27,754 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1679273787748, max timestamp=1679273787752Num uniques 1026; 2023-03-20 00:56:27,754 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=10346, suffix heap size=134672,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=226 2023-03-20 00:56:27,754 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1679273787748, max timestamp=1679273787752Num uniques 1026; 2023-03-20 00:56:27,754 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-03-20 00:56:27,754 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1679273787748, max timestamp=1679273787752Num uniques 1026; 2023-03-20 00:56:27,754 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=10346, suffix heap size=134672,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=226 2023-03-20 00:56:27,767 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:27,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48422 deadline: 1679273847765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:27,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:27,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48368 deadline: 1679273847766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:27,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1679273787763, max timestamp=1679273787766Num uniques 1085; 2023-03-20 00:56:27,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1679273787763, max timestamp=1679273787766Num uniques 1085; 2023-03-20 00:56:27,769 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=8470, suffix heap size=140808,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=185 2023-03-20 00:56:27,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=8470, suffix heap size=140808,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=185 2023-03-20 00:56:27,769 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8464, totalHeapSize=19480, min timestamp=1679273787763, max timestamp=1679273787766Num uniques 1035; 2023-03-20 00:56:27,769 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47394, new segment data size=8464, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-03-20 00:56:27,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1679273787778, max timestamp=1679273787782Num uniques 1044; 2023-03-20 00:56:27,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-03-20 00:56:27,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1679273787778, max timestamp=1679273787782Num uniques 1044; 2023-03-20 00:56:27,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-03-20 00:56:27,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1679273787778, max timestamp=1679273787782Num uniques 1044; 2023-03-20 00:56:27,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-03-20 00:56:27,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679273787800, max timestamp=1679273787805Num uniques 1053; 2023-03-20 00:56:27,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-03-20 00:56:27,808 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679273787800, max timestamp=1679273787805Num uniques 1053; 2023-03-20 00:56:27,808 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-03-20 00:56:27,808 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679273787800, max timestamp=1679273787805Num uniques 1053; 2023-03-20 00:56:27,808 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-03-20 00:56:27,827 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:27,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48408 deadline: 1679273847827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:27,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679273787819, max timestamp=1679273787825Num uniques 1012; 2023-03-20 00:56:27,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=6870, suffix heap size=133216,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=150 2023-03-20 00:56:27,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679273787819, max timestamp=1679273787825Num uniques 1012; 2023-03-20 00:56:27,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679273787819, max timestamp=1679273787825Num uniques 1012; 2023-03-20 00:56:27,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=6870, suffix heap size=133216,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=150 2023-03-20 00:56:27,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=6870, suffix heap size=133216,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=150 2023-03-20 00:56:27,849 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1679273787843, max timestamp=1679273787847Num uniques 1009; 2023-03-20 00:56:27,849 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=7826, suffix heap size=132904,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=171 2023-03-20 00:56:27,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679273787843, max timestamp=1679273787847Num uniques 1009; 2023-03-20 00:56:27,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-20 00:56:27,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679273787843, max timestamp=1679273787847Num uniques 1009; 2023-03-20 00:56:27,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-20 00:56:27,871 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1679273787862, max timestamp=1679273787870Num uniques 1009; 2023-03-20 00:56:27,871 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1679273787862, max timestamp=1679273787870Num uniques 1009; 2023-03-20 00:56:27,871 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1679273787862, max timestamp=1679273787870Num uniques 1030; 2023-03-20 00:56:27,871 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8240, suffix heap size=132904,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=180 2023-03-20 00:56:27,871 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10530, suffix heap size=132904,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=230 2023-03-20 00:56:27,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=10530, suffix heap size=135088,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=230 2023-03-20 00:56:27,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1679273787879, max timestamp=1679273787882Num uniques 1039; 2023-03-20 00:56:27,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=10944, suffix heap size=136024,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=239 2023-03-20 00:56:27,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1679273787879, max timestamp=1679273787882Num uniques 1089; 2023-03-20 00:56:27,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=10944, suffix heap size=141224,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=239 2023-03-20 00:56:27,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1679273787879, max timestamp=1679273787882Num uniques 1089; 2023-03-20 00:56:27,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=10944, suffix heap size=141224,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=239 2023-03-20 00:56:27,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:27,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:27,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48422 deadline: 1679273847896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:27,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48428 deadline: 1679273847895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:27,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9058, totalHeapSize=20832, min timestamp=1679273787888, max timestamp=1679273787897Num uniques 1098; 2023-03-20 00:56:27,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9058, totalHeapSize=20832, min timestamp=1679273787888, max timestamp=1679273787897Num uniques 1098; 2023-03-20 00:56:27,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=50278, new segment data size=9058, suffix heap size=142160,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=198 2023-03-20 00:56:27,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=50278, new segment data size=9058, suffix heap size=142160,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=198 2023-03-20 00:56:27,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9058, totalHeapSize=20832, min timestamp=1679273787888, max timestamp=1679273787897Num uniques 1098; 2023-03-20 00:56:27,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=50278, new segment data size=9058, suffix heap size=142160,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=198 2023-03-20 00:56:27,905 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/b96464e5dfc8454190814d7d4215e006 2023-03-20 00:56:27,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1679273787919, max timestamp=1679273787928Num uniques 1057; 2023-03-20 00:56:27,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=7185, suffix heap size=137896,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=157 2023-03-20 00:56:27,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1679273787919, max timestamp=1679273787932Num uniques 1057; 2023-03-20 00:56:27,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=7185, suffix heap size=137896,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=157 2023-03-20 00:56:27,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=151, cellSize=6915, totalHeapSize=15944, min timestamp=1679273787919, max timestamp=1679273787932Num uniques 1057; 2023-03-20 00:56:27,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=6915, suffix heap size=137896,new segment heap size=15944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=151 2023-03-20 00:56:27,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7596, totalHeapSize=17504, min timestamp=1679273787969, max timestamp=1679273787972Num uniques 1016; 2023-03-20 00:56:27,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46526, new segment data size=7596, suffix heap size=133632,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1016, new segment cells count=166 2023-03-20 00:56:27,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7596, totalHeapSize=17504, min timestamp=1679273787969, max timestamp=1679273787972Num uniques 1016; 2023-03-20 00:56:27,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46526, new segment data size=7596, suffix heap size=133632,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1016, new segment cells count=166 2023-03-20 00:56:27,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7596, totalHeapSize=17504, min timestamp=1679273787969, max timestamp=1679273787972Num uniques 1010; 2023-03-20 00:56:27,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46256, new segment data size=7596, suffix heap size=133008,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1010, new segment cells count=166 2023-03-20 00:56:27,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1679273787988, max timestamp=1679273787993Num uniques 1025; 2023-03-20 00:56:27,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46940, new segment data size=10300, suffix heap size=134568,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1025, new segment cells count=225 2023-03-20 00:56:27,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679273787988, max timestamp=1679273787993Num uniques 1025; 2023-03-20 00:56:27,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46940, new segment data size=6870, suffix heap size=134568,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1025, new segment cells count=150 2023-03-20 00:56:27,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679273787988, max timestamp=1679273787993Num uniques 1025; 2023-03-20 00:56:27,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46940, new segment data size=6870, suffix heap size=134568,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1025, new segment cells count=150 2023-03-20 00:56:28,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=184, cellSize=8414, totalHeapSize=19376, min timestamp=1679273788008, max timestamp=1679273788016Num uniques 1084; 2023-03-20 00:56:28,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=184, cellSize=8422, totalHeapSize=19376, min timestamp=1679273788008, max timestamp=1679273788016Num uniques 1009; 2023-03-20 00:56:28,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49634, new segment data size=8414, suffix heap size=140704,new segment heap size=19376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=184 2023-03-20 00:56:28,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46212, new segment data size=8422, suffix heap size=132904,new segment heap size=19376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=184 2023-03-20 00:56:28,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=184, cellSize=8414, totalHeapSize=19376, min timestamp=1679273788008, max timestamp=1679273788016Num uniques 1009; 2023-03-20 00:56:28,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46204, new segment data size=8414, suffix heap size=132904,new segment heap size=19376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=184 2023-03-20 00:56:28,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11128, totalHeapSize=25512, min timestamp=1679273788029, max timestamp=1679273788033Num uniques 1043; 2023-03-20 00:56:28,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47768, new segment data size=11128, suffix heap size=136440,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1043, new segment cells count=243 2023-03-20 00:56:28,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11128, totalHeapSize=25512, min timestamp=1679273788029, max timestamp=1679273788033Num uniques 1043; 2023-03-20 00:56:28,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47768, new segment data size=11128, suffix heap size=136440,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1043, new segment cells count=243 2023-03-20 00:56:28,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679273788029, max timestamp=1679273788033Num uniques 1043; 2023-03-20 00:56:28,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47768, new segment data size=6870, suffix heap size=136440,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1043, new segment cells count=150 2023-03-20 00:56:28,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:28,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273848071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=152, cellSize=6960, totalHeapSize=16048, min timestamp=1679273788040, max timestamp=1679273788073Num uniques 1102; 2023-03-20 00:56:28,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=152, cellSize=6960, totalHeapSize=16048, min timestamp=1679273788040, max timestamp=1679273788073Num uniques 1102; 2023-03-20 00:56:28,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=6960, suffix heap size=142576,new segment heap size=16048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=152 2023-03-20 00:56:28,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=6960, suffix heap size=142576,new segment heap size=16048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=152 2023-03-20 00:56:28,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679273788040, max timestamp=1679273788073Num uniques 1009; 2023-03-20 00:56:28,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46212, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-20 00:56:28,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=161, cellSize=7366, totalHeapSize=16984, min timestamp=1679273788087, max timestamp=1679273788089Num uniques 1011; 2023-03-20 00:56:28,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=161, cellSize=7366, totalHeapSize=16984, min timestamp=1679273788087, max timestamp=1679273788089Num uniques 1011; 2023-03-20 00:56:28,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=161, cellSize=7366, totalHeapSize=16984, min timestamp=1679273788087, max timestamp=1679273788089Num uniques 1009; 2023-03-20 00:56:28,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46296, new segment data size=7366, suffix heap size=133112,new segment heap size=16984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1011, new segment cells count=161 2023-03-20 00:56:28,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46206, new segment data size=7366, suffix heap size=132904,new segment heap size=16984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=161 2023-03-20 00:56:28,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46296, new segment data size=7366, suffix heap size=133112,new segment heap size=16984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1011, new segment cells count=161 2023-03-20 00:56:28,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=220, cellSize=10070, totalHeapSize=23120, min timestamp=1679273788101, max timestamp=1679273788104Num uniques 1020; 2023-03-20 00:56:28,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46710, new segment data size=10070, suffix heap size=134048,new segment heap size=23120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1020, new segment cells count=220 2023-03-20 00:56:28,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=170, cellSize=7780, totalHeapSize=17920, min timestamp=1679273788102, max timestamp=1679273788104Num uniques 1020; 2023-03-20 00:56:28,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46710, new segment data size=7780, suffix heap size=134048,new segment heap size=17920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1020, new segment cells count=170 2023-03-20 00:56:28,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=170, cellSize=7780, totalHeapSize=17920, min timestamp=1679273788102, max timestamp=1679273788104Num uniques 1020; 2023-03-20 00:56:28,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46710, new segment data size=7780, suffix heap size=134048,new segment heap size=17920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1020, new segment cells count=170 2023-03-20 00:56:28,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8194, totalHeapSize=18856, min timestamp=1679273788116, max timestamp=1679273788119Num uniques 1029; 2023-03-20 00:56:28,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47124, new segment data size=8194, suffix heap size=134984,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1029, new segment cells count=179 2023-03-20 00:56:28,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8194, totalHeapSize=18856, min timestamp=1679273788116, max timestamp=1679273788119Num uniques 1079; 2023-03-20 00:56:28,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49414, new segment data size=8194, suffix heap size=140184,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1079, new segment cells count=179 2023-03-20 00:56:28,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8194, totalHeapSize=18856, min timestamp=1679273788116, max timestamp=1679273788119Num uniques 1029; 2023-03-20 00:56:28,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47124, new segment data size=8194, suffix heap size=134984,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1029, new segment cells count=179 2023-03-20 00:56:28,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10898, totalHeapSize=24992, min timestamp=1679273788166, max timestamp=1679273788172Num uniques 1038; 2023-03-20 00:56:28,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47538, new segment data size=10898, suffix heap size=135920,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=238 2023-03-20 00:56:28,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10898, totalHeapSize=24992, min timestamp=1679273788166, max timestamp=1679273788172Num uniques 1038; 2023-03-20 00:56:28,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47538, new segment data size=10898, suffix heap size=135920,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=238 2023-03-20 00:56:28,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10898, totalHeapSize=24992, min timestamp=1679273788166, max timestamp=1679273788172Num uniques 1038; 2023-03-20 00:56:28,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47538, new segment data size=10898, suffix heap size=135920,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=238 2023-03-20 00:56:28,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=247, cellSize=11312, totalHeapSize=25928, min timestamp=1679273788188, max timestamp=1679273788195Num uniques 1097; 2023-03-20 00:56:28,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=197, cellSize=9022, totalHeapSize=20728, min timestamp=1679273788188, max timestamp=1679273788196Num uniques 1097; 2023-03-20 00:56:28,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=50242, new segment data size=11312, suffix heap size=142056,new segment heap size=25928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=247 2023-03-20 00:56:28,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=50242, new segment data size=9022, suffix heap size=142056,new segment heap size=20728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=197 2023-03-20 00:56:28,200 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=197, cellSize=9012, totalHeapSize=20728, min timestamp=1679273788188, max timestamp=1679273788196Num uniques 1097; 2023-03-20 00:56:28,200 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=50232, new segment data size=9012, suffix heap size=142056,new segment heap size=20728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=197 2023-03-20 00:56:28,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=206, cellSize=9430, totalHeapSize=21664, min timestamp=1679273788209, max timestamp=1679273788213Num uniques 1106; 2023-03-20 00:56:28,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=206, cellSize=9430, totalHeapSize=21664, min timestamp=1679273788209, max timestamp=1679273788213Num uniques 1056; 2023-03-20 00:56:28,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=206, cellSize=9430, totalHeapSize=21664, min timestamp=1679273788209, max timestamp=1679273788213Num uniques 1056; 2023-03-20 00:56:28,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=48360, new segment data size=9430, suffix heap size=137792,new segment heap size=21664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1056, new segment cells count=206 2023-03-20 00:56:28,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=50650, new segment data size=9430, suffix heap size=142992,new segment heap size=21664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1106, new segment cells count=206 2023-03-20 00:56:28,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48360, new segment data size=9430, suffix heap size=137792,new segment heap size=21664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1056, new segment cells count=206 2023-03-20 00:56:28,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:28,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48422 deadline: 1679273848225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:28,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273848227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=215, cellSize=9840, totalHeapSize=22600, min timestamp=1679273788219, max timestamp=1679273788227Num uniques 1065; 2023-03-20 00:56:28,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48770, new segment data size=9840, suffix heap size=138728,new segment heap size=22600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1065, new segment cells count=215 2023-03-20 00:56:28,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=215, cellSize=9834, totalHeapSize=22600, min timestamp=1679273788219, max timestamp=1679273788229Num uniques 1065; 2023-03-20 00:56:28,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=48764, new segment data size=9834, suffix heap size=138728,new segment heap size=22600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1065, new segment cells count=215 2023-03-20 00:56:28,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679273788219, max timestamp=1679273788227Num uniques 1065; 2023-03-20 00:56:28,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=48770, new segment data size=6870, suffix heap size=138728,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1065, new segment cells count=150 2023-03-20 00:56:28,254 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:28,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273848252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,255 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=224, cellSize=10254, totalHeapSize=23536, min timestamp=1679273788246, max timestamp=1679273788253Num uniques 1074; 2023-03-20 00:56:28,255 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=49184, new segment data size=10254, suffix heap size=139664,new segment heap size=23536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1074, new segment cells count=224 2023-03-20 00:56:28,255 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=224, cellSize=10254, totalHeapSize=23536, min timestamp=1679273788246, max timestamp=1679273788253Num uniques 1074; 2023-03-20 00:56:28,255 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49184, new segment data size=10254, suffix heap size=139664,new segment heap size=23536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1074, new segment cells count=224 2023-03-20 00:56:28,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679273788248, max timestamp=1679273788253Num uniques 1009; 2023-03-20 00:56:28,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-20 00:56:28,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=233, cellSize=10668, totalHeapSize=24472, min timestamp=1679273788264, max timestamp=1679273788270Num uniques 1083; 2023-03-20 00:56:28,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49598, new segment data size=10668, suffix heap size=140600,new segment heap size=24472 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1083, new segment cells count=233 2023-03-20 00:56:28,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=233, cellSize=10668, totalHeapSize=24472, min timestamp=1679273788264, max timestamp=1679273788270Num uniques 1083; 2023-03-20 00:56:28,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=233, cellSize=10668, totalHeapSize=24472, min timestamp=1679273788264, max timestamp=1679273788270Num uniques 1009; 2023-03-20 00:56:28,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49598, new segment data size=10668, suffix heap size=140600,new segment heap size=24472 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1083, new segment cells count=233 2023-03-20 00:56:28,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10668, suffix heap size=132904,new segment heap size=24472 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=233 2023-03-20 00:56:28,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=192, cellSize=8792, totalHeapSize=20208, min timestamp=1679273788285, max timestamp=1679273788289Num uniques 1092; 2023-03-20 00:56:28,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=192, cellSize=8792, totalHeapSize=20208, min timestamp=1679273788285, max timestamp=1679273788289Num uniques 1092; 2023-03-20 00:56:28,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=192, cellSize=8792, totalHeapSize=20208, min timestamp=1679273788285, max timestamp=1679273788289Num uniques 1092; 2023-03-20 00:56:28,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=50012, new segment data size=8792, suffix heap size=141536,new segment heap size=20208 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1092, new segment cells count=192 2023-03-20 00:56:28,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50012, new segment data size=8792, suffix heap size=141536,new segment heap size=20208 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1092, new segment cells count=192 2023-03-20 00:56:28,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=50012, new segment data size=8792, suffix heap size=141536,new segment heap size=20208 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1092, new segment cells count=192 2023-03-20 00:56:28,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=151, cellSize=6915, totalHeapSize=15944, min timestamp=1679273788295, max timestamp=1679273788307Num uniques 1051; 2023-03-20 00:56:28,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=48135, new segment data size=6915, suffix heap size=137272,new segment heap size=15944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1051, new segment cells count=151 2023-03-20 00:56:28,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=151, cellSize=6915, totalHeapSize=15944, min timestamp=1679273788295, max timestamp=1679273788307Num uniques 1051; 2023-03-20 00:56:28,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=151, cellSize=6915, totalHeapSize=15944, min timestamp=1679273788295, max timestamp=1679273788307Num uniques 1051; 2023-03-20 00:56:28,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48135, new segment data size=6915, suffix heap size=137272,new segment heap size=15944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1051, new segment cells count=151 2023-03-20 00:56:28,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=48135, new segment data size=6915, suffix heap size=137272,new segment heap size=15944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1051, new segment cells count=151 2023-03-20 00:56:28,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=160, cellSize=7320, totalHeapSize=16880, min timestamp=1679273788322, max timestamp=1679273788328Num uniques 1010; 2023-03-20 00:56:28,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=160, cellSize=7320, totalHeapSize=16880, min timestamp=1679273788322, max timestamp=1679273788328Num uniques 1010; 2023-03-20 00:56:28,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46250, new segment data size=7320, suffix heap size=133008,new segment heap size=16880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1010, new segment cells count=160 2023-03-20 00:56:28,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679273788322, max timestamp=1679273788328Num uniques 1010; 2023-03-20 00:56:28,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46250, new segment data size=6870, suffix heap size=133008,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1010, new segment cells count=150 2023-03-20 00:56:28,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46250, new segment data size=7320, suffix heap size=133008,new segment heap size=16880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1010, new segment cells count=160 2023-03-20 00:56:28,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=169, cellSize=7734, totalHeapSize=17816, min timestamp=1679273788344, max timestamp=1679273788347Num uniques 1009; 2023-03-20 00:56:28,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=7734, suffix heap size=132904,new segment heap size=17816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=169 2023-03-20 00:56:28,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=219, cellSize=10024, totalHeapSize=23016, min timestamp=1679273788340, max timestamp=1679273788347Num uniques 1019; 2023-03-20 00:56:28,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46664, new segment data size=10024, suffix heap size=133944,new segment heap size=23016 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1019, new segment cells count=219 2023-03-20 00:56:28,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=219, cellSize=10024, totalHeapSize=23016, min timestamp=1679273788340, max timestamp=1679273788347Num uniques 1019; 2023-03-20 00:56:28,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46664, new segment data size=10024, suffix heap size=133944,new segment heap size=23016 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1019, new segment cells count=219 2023-03-20 00:56:28,357 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/ab9d24979c6644e09cceec5bf045d38e 2023-03-20 00:56:28,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679273788354, max timestamp=1679273788360Num uniques 1078; 2023-03-20 00:56:28,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49368, new segment data size=6870, suffix heap size=140080,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1078, new segment cells count=150 2023-03-20 00:56:28,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679273788354, max timestamp=1679273788360Num uniques 1028; 2023-03-20 00:56:28,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679273788354, max timestamp=1679273788360Num uniques 1078; 2023-03-20 00:56:28,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47078, new segment data size=6870, suffix heap size=134880,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1028, new segment cells count=150 2023-03-20 00:56:28,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49368, new segment data size=6870, suffix heap size=140080,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1078, new segment cells count=150 2023-03-20 00:56:28,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=237, cellSize=10852, totalHeapSize=24888, min timestamp=1679273788374, max timestamp=1679273788380Num uniques 1009; 2023-03-20 00:56:28,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10852, suffix heap size=132904,new segment heap size=24888 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=237 2023-03-20 00:56:28,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679273788374, max timestamp=1679273788380Num uniques 1009; 2023-03-20 00:56:28,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-20 00:56:28,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679273788374, max timestamp=1679273788380Num uniques 1009; 2023-03-20 00:56:28,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-20 00:56:28,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=246, cellSize=11266, totalHeapSize=25824, min timestamp=1679273788406, max timestamp=1679273788411Num uniques 1096; 2023-03-20 00:56:28,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=50196, new segment data size=11266, suffix heap size=141952,new segment heap size=25824 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1096, new segment cells count=246 2023-03-20 00:56:28,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=246, cellSize=11266, totalHeapSize=25824, min timestamp=1679273788406, max timestamp=1679273788411Num uniques 1009; 2023-03-20 00:56:28,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=246, cellSize=11266, totalHeapSize=25824, min timestamp=1679273788406, max timestamp=1679273788411Num uniques 1009; 2023-03-20 00:56:28,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=11266, suffix heap size=132904,new segment heap size=25824 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=246 2023-03-20 00:56:28,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=11266, suffix heap size=132904,new segment heap size=25824 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=246 2023-03-20 00:56:28,424 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:28,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48422 deadline: 1679273848422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=155, cellSize=7095, totalHeapSize=16360, min timestamp=1679273788420, max timestamp=1679273788423Num uniques 1105; 2023-03-20 00:56:28,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=50605, new segment data size=7095, suffix heap size=142888,new segment heap size=16360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1105, new segment cells count=155 2023-03-20 00:56:28,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=155, cellSize=7095, totalHeapSize=16360, min timestamp=1679273788420, max timestamp=1679273788423Num uniques 1105; 2023-03-20 00:56:28,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=155, cellSize=7095, totalHeapSize=16360, min timestamp=1679273788420, max timestamp=1679273788423Num uniques 1105; 2023-03-20 00:56:28,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=50605, new segment data size=7095, suffix heap size=142888,new segment heap size=16360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1105, new segment cells count=155 2023-03-20 00:56:28,426 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=50605, new segment data size=7095, suffix heap size=142888,new segment heap size=16360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1105, new segment cells count=155 2023-03-20 00:56:28,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=214, cellSize=9794, totalHeapSize=22496, min timestamp=1679273788429, max timestamp=1679273788436Num uniques 1014; 2023-03-20 00:56:28,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=214, cellSize=9794, totalHeapSize=22496, min timestamp=1679273788429, max timestamp=1679273788436Num uniques 1014; 2023-03-20 00:56:28,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46434, new segment data size=9794, suffix heap size=133424,new segment heap size=22496 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1014, new segment cells count=214 2023-03-20 00:56:28,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46434, new segment data size=9794, suffix heap size=133424,new segment heap size=22496 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1014, new segment cells count=214 2023-03-20 00:56:28,439 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=214, cellSize=9794, totalHeapSize=22496, min timestamp=1679273788429, max timestamp=1679273788436Num uniques 1014; 2023-03-20 00:56:28,439 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46434, new segment data size=9794, suffix heap size=133424,new segment heap size=22496 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1014, new segment cells count=214 2023-03-20 00:56:28,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=223, cellSize=10208, totalHeapSize=23432, min timestamp=1679273788445, max timestamp=1679273788447Num uniques 1073; 2023-03-20 00:56:28,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49138, new segment data size=10208, suffix heap size=139560,new segment heap size=23432 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1073, new segment cells count=223 2023-03-20 00:56:28,450 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=223, cellSize=10208, totalHeapSize=23432, min timestamp=1679273788445, max timestamp=1679273788447Num uniques 1073; 2023-03-20 00:56:28,450 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49138, new segment data size=10208, suffix heap size=139560,new segment heap size=23432 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1073, new segment cells count=223 2023-03-20 00:56:28,450 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=223, cellSize=10208, totalHeapSize=23432, min timestamp=1679273788445, max timestamp=1679273788447Num uniques 1073; 2023-03-20 00:56:28,450 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49138, new segment data size=10208, suffix heap size=139560,new segment heap size=23432 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1073, new segment cells count=223 2023-03-20 00:56:28,461 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=332, cellSize=15202, totalHeapSize=34768, min timestamp=1679273788456, max timestamp=1679273788459Num uniques 1082; 2023-03-20 00:56:28,461 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=49552, new segment data size=15202, suffix heap size=140496,new segment heap size=34768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1082, new segment cells count=332 2023-03-20 00:56:28,461 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=332, cellSize=15202, totalHeapSize=34768, min timestamp=1679273788456, max timestamp=1679273788459Num uniques 1082; 2023-03-20 00:56:28,461 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49552, new segment data size=15202, suffix heap size=140496,new segment heap size=34768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1082, new segment cells count=332 2023-03-20 00:56:28,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=232, cellSize=10622, totalHeapSize=24368, min timestamp=1679273788456, max timestamp=1679273788459Num uniques 1082; 2023-03-20 00:56:28,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49552, new segment data size=10622, suffix heap size=140496,new segment heap size=24368 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1082, new segment cells count=232 2023-03-20 00:56:28,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:28,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273848472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=241, cellSize=11036, totalHeapSize=25304, min timestamp=1679273788466, max timestamp=1679273788473Num uniques 1191; 2023-03-20 00:56:28,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=54546, new segment data size=11036, suffix heap size=151832,new segment heap size=25304 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1191, new segment cells count=241 2023-03-20 00:56:28,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=191, cellSize=8746, totalHeapSize=20104, min timestamp=1679273788469, max timestamp=1679273788473Num uniques 1191; 2023-03-20 00:56:28,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=54546, new segment data size=8746, suffix heap size=151832,new segment heap size=20104 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1191, new segment cells count=191 2023-03-20 00:56:28,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679273788469, max timestamp=1679273788473Num uniques 1091; 2023-03-20 00:56:28,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49966, new segment data size=6870, suffix heap size=141432,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1091, new segment cells count=150 2023-03-20 00:56:28,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=200, cellSize=9160, totalHeapSize=21040, min timestamp=1679273788477, max timestamp=1679273788486Num uniques 1050; 2023-03-20 00:56:28,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48090, new segment data size=9160, suffix heap size=137168,new segment heap size=21040 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1050, new segment cells count=200 2023-03-20 00:56:28,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=200, cellSize=9160, totalHeapSize=21040, min timestamp=1679273788477, max timestamp=1679273788486Num uniques 1100; 2023-03-20 00:56:28,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=50380, new segment data size=9160, suffix heap size=142368,new segment heap size=21040 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1100, new segment cells count=200 2023-03-20 00:56:28,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=200, cellSize=9150, totalHeapSize=21040, min timestamp=1679273788477, max timestamp=1679273788487Num uniques 1009; 2023-03-20 00:56:28,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46204, new segment data size=9150, suffix heap size=132904,new segment heap size=21040 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=200 2023-03-20 00:56:28,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679273788500, max timestamp=1679273788504Num uniques 1059; 2023-03-20 00:56:28,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48495, new segment data size=7275, suffix heap size=138104,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1059, new segment cells count=159 2023-03-20 00:56:28,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679273788500, max timestamp=1679273788504Num uniques 1059; 2023-03-20 00:56:28,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679273788500, max timestamp=1679273788504Num uniques 1059; 2023-03-20 00:56:28,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48495, new segment data size=7275, suffix heap size=138104,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1059, new segment cells count=159 2023-03-20 00:56:28,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48495, new segment data size=7275, suffix heap size=138104,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1059, new segment cells count=159 2023-03-20 00:56:28,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679273788516, max timestamp=1679273788519Num uniques 1018; 2023-03-20 00:56:28,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-20 00:56:28,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679273788516, max timestamp=1679273788519Num uniques 1018; 2023-03-20 00:56:28,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-20 00:56:28,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679273788516, max timestamp=1679273788519Num uniques 1018; 2023-03-20 00:56:28,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-20 00:56:28,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679273788525, max timestamp=1679273788532Num uniques 1027; 2023-03-20 00:56:28,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-20 00:56:28,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679273788525, max timestamp=1679273788532Num uniques 1027; 2023-03-20 00:56:28,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-20 00:56:28,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679273788530, max timestamp=1679273788532Num uniques 1027; 2023-03-20 00:56:28,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-20 00:56:28,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679273788542, max timestamp=1679273788549Num uniques 1086; 2023-03-20 00:56:28,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-03-20 00:56:28,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679273788542, max timestamp=1679273788549Num uniques 1036; 2023-03-20 00:56:28,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-03-20 00:56:28,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:28,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679273788542, max timestamp=1679273788549Num uniques 1086; 2023-03-20 00:56:28,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-03-20 00:56:28,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273848550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:28,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273848576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679273788571, max timestamp=1679273788577Num uniques 1095; 2023-03-20 00:56:28,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=50140, new segment data size=7642, suffix heap size=141848,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=167 2023-03-20 00:56:28,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8920, totalHeapSize=20520, min timestamp=1679273788571, max timestamp=1679273788577Num uniques 1095; 2023-03-20 00:56:28,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=50140, new segment data size=8920, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-03-20 00:56:28,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679273788571, max timestamp=1679273788577Num uniques 1095; 2023-03-20 00:56:28,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-03-20 00:56:28,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1679273788594, max timestamp=1679273788605Num uniques 1054; 2023-03-20 00:56:28,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-03-20 00:56:28,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1679273788594, max timestamp=1679273788605Num uniques 1026; 2023-03-20 00:56:28,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1679273788594, max timestamp=1679273788605Num uniques 1054; 2023-03-20 00:56:28,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46992, new segment data size=9340, suffix heap size=134672,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=204 2023-03-20 00:56:28,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-03-20 00:56:28,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679273788619, max timestamp=1679273788636Num uniques 1063; 2023-03-20 00:56:28,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-03-20 00:56:28,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679273788619, max timestamp=1679273788636Num uniques 1063; 2023-03-20 00:56:28,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-03-20 00:56:28,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679273788619, max timestamp=1679273788636Num uniques 1063; 2023-03-20 00:56:28,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-03-20 00:56:28,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679273788656, max timestamp=1679273788659Num uniques 1022; 2023-03-20 00:56:28,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-03-20 00:56:28,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679273788656, max timestamp=1679273788659Num uniques 1022; 2023-03-20 00:56:28,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-03-20 00:56:28,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679273788656, max timestamp=1679273788660Num uniques 1022; 2023-03-20 00:56:28,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-03-20 00:56:28,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1679273788664, max timestamp=1679273788677Num uniques 1031; 2023-03-20 00:56:28,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=12866, suffix heap size=135192,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=281 2023-03-20 00:56:28,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679273788664, max timestamp=1679273788677Num uniques 1031; 2023-03-20 00:56:28,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-03-20 00:56:28,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1679273788664, max timestamp=1679273788677Num uniques 1031; 2023-03-20 00:56:28,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=12866, suffix heap size=135192,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=281 2023-03-20 00:56:28,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1679273788694, max timestamp=1679273788698Num uniques 1140; 2023-03-20 00:56:28,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=10990, suffix heap size=146528,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=240 2023-03-20 00:56:28,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1679273788694, max timestamp=1679273788698Num uniques 1140; 2023-03-20 00:56:28,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=10990, suffix heap size=146528,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=240 2023-03-20 00:56:28,701 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679273788694, max timestamp=1679273788698Num uniques 1040; 2023-03-20 00:56:28,701 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-03-20 00:56:28,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679273788713, max timestamp=1679273788716Num uniques 1049; 2023-03-20 00:56:28,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=6870, suffix heap size=137064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=150 2023-03-20 00:56:28,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1679273788713, max timestamp=1679273788716Num uniques 1099; 2023-03-20 00:56:28,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=13694, suffix heap size=142264,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=299 2023-03-20 00:56:28,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1679273788713, max timestamp=1679273788716Num uniques 1099; 2023-03-20 00:56:28,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=13694, suffix heap size=142264,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=299 2023-03-20 00:56:28,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1679273788727, max timestamp=1679273788732Num uniques 1158; 2023-03-20 00:56:28,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=9520, suffix heap size=148400,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=208 2023-03-20 00:56:28,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679273788727, max timestamp=1679273788732Num uniques 1158; 2023-03-20 00:56:28,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=6870, suffix heap size=148400,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=150 2023-03-20 00:56:28,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679273788727, max timestamp=1679273788732Num uniques 1009; 2023-03-20 00:56:28,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46206, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-20 00:56:28,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7635, totalHeapSize=17608, min timestamp=1679273788742, max timestamp=1679273788748Num uniques 1067; 2023-03-20 00:56:28,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7640, totalHeapSize=17608, min timestamp=1679273788742, max timestamp=1679273788748Num uniques 1009; 2023-03-20 00:56:28,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48855, new segment data size=7635, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-03-20 00:56:28,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46210, new segment data size=7640, suffix heap size=132904,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=167 2023-03-20 00:56:28,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679273788742, max timestamp=1679273788748Num uniques 1009; 2023-03-20 00:56:28,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46212, new segment data size=7642, suffix heap size=132904,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=167 2023-03-20 00:56:28,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1679273788756, max timestamp=1679273788761Num uniques 1026; 2023-03-20 00:56:28,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1679273788756, max timestamp=1679273788761Num uniques 1026; 2023-03-20 00:56:28,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1679273788756, max timestamp=1679273788761Num uniques 1026; 2023-03-20 00:56:28,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-03-20 00:56:28,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-03-20 00:56:28,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-03-20 00:56:28,769 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/da23df92d8404f5d955401c04ae9374d 2023-03-20 00:56:28,774 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/b96464e5dfc8454190814d7d4215e006 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/b96464e5dfc8454190814d7d4215e006 2023-03-20 00:56:28,778 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/b96464e5dfc8454190814d7d4215e006, entries=50, sequenceid=8, filesize=7.0 K 2023-03-20 00:56:28,779 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/ab9d24979c6644e09cceec5bf045d38e as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/ab9d24979c6644e09cceec5bf045d38e 2023-03-20 00:56:28,788 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/ab9d24979c6644e09cceec5bf045d38e, entries=50, sequenceid=8, filesize=7.0 K 2023-03-20 00:56:28,788 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/da23df92d8404f5d955401c04ae9374d as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/da23df92d8404f5d955401c04ae9374d 2023-03-20 00:56:28,796 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/da23df92d8404f5d955401c04ae9374d, entries=50, sequenceid=8, filesize=7.0 K 2023-03-20 00:56:28,801 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=120.76 KB/123660 for 51e645e88856df580ad0de225d47b16a in 1347ms, sequenceid=8, compaction requested=false 2023-03-20 00:56:28,801 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-20 00:56:28,802 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:28,803 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:28,803 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:28,803 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:28,803 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,803 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:28,803 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:28,804 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:28,804 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:28,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:28,804 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:28,804 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:28,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:28,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:28,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:28,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:28,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:28,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:28,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@607e356c[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:28,807 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51e645e88856df580ad0de225d47b16a 3/3 column families, dataSize=134.18 KB heapSize=383.39 KB 2023-03-20 00:56:28,807 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:28,807 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:28,808 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=A 2023-03-20 00:56:28,808 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-20 00:56:28,808 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=B 2023-03-20 00:56:28,808 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-20 00:56:28,808 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=C 2023-03-20 00:56:28,808 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-20 00:56:28,810 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:28,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:28,811 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:28,811 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:28,811 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:28,815 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:28,815 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:28,815 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:28,822 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,822 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,822 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:28,825 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:28,825 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:28,825 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:28,825 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:28,825 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:28,826 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:28,826 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:28,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:28,826 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:28,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:28,826 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,828 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,830 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:28,830 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:28,830 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:28,830 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:28,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:28,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:28,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:28,830 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:28,830 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:28,830 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:28,830 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:28,830 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,830 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:28,830 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:28,830 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:28,843 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:28,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273848840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,843 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:28,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273848841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:28,846 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:28,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273848846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273848846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:28,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273848850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,930 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:28,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:28,946 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:28,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:28,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273848944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:28,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:28,948 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:28,948 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:28,948 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:28,948 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:28,949 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:28,949 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:28,949 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,950 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,950 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:28,950 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:28,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:28,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:28,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273848947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:28,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273848947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:28,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:28,953 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:28,953 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:28,953 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:28,953 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:28,953 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:28,953 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:28,953 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:28,953 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:28,953 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:28,953 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:28,953 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:28,953 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:28,953 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:28,953 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,954 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:28,954 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:28,955 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:28,955 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:28,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:28,955 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:28,955 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:28,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:28,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:28,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b45aab4[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:28,957 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:28,957 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:28,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:28,958 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:28,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273848944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:28,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:28,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:28,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273848954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,958 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:28,959 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,959 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:28,960 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:28,960 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:28,962 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:28,962 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:28,962 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:28,962 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:28,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:28,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:28,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:28,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:28,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:28,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:28,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:28,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:28,964 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:28,964 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:28,964 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:28,964 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:28,964 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:28,965 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:28,965 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:28,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:28,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:28,965 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:28,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:28,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:28,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:28,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:28,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:28,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:28,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:28,965 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,967 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:28,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:28,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,968 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,968 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:28,969 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:28,969 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:28,969 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:28,969 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:28,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:28,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:28,969 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:28,969 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:28,969 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:28,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:28,969 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:28,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:28,969 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:28,969 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:28,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:28,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:29,069 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:29,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:29,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:29,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:29,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:29,075 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:29,075 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:29,075 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:29,075 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:29,075 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,075 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:29,076 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,076 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:29,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:29,076 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,076 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:29,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:29,077 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:29,078 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:29,078 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:29,078 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:29,078 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:29,078 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:29,082 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:29,082 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:29,082 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:29,082 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:29,082 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:29,082 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:29,082 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:29,082 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:29,082 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,083 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:29,083 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:29,083 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,084 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:29,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:29,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:29,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:29,085 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,085 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:29,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:29,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:29,086 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:29,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ec81285[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:29,086 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,087 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,087 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:29,087 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:29,087 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,087 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:29,087 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:29,088 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,088 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:29,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:29,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:29,088 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:29,088 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:29,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:29,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:29,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:29,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:29,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:29,090 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:29,090 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:29,090 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:29,090 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:29,090 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:29,091 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,091 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:29,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:29,091 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:29,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:29,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:29,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,092 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,092 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:29,094 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:29,094 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,094 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:29,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:29,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:29,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:29,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:29,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:29,094 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:29,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:29,094 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:29,095 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:29,095 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:29,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:29,095 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:29,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:29,095 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,095 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,095 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,095 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:29,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:29,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273849148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:29,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273849151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:29,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273849151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:29,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273849159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:29,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273849160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,195 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:29,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:29,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:29,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:29,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:29,198 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:29,198 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:29,198 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:29,198 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:29,198 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,198 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:29,199 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:29,199 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:29,199 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,200 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:29,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:29,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:29,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:29,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:29,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:29,200 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:29,200 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:29,200 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:29,201 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:29,201 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:29,201 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:29,201 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:29,201 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:29,201 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:29,201 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:29,201 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:29,201 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,201 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:29,201 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:29,202 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,202 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:29,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:29,203 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:29,203 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:29,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:29,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:29,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:29,204 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:29,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36746001[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:29,204 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,205 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,205 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:29,205 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:29,205 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:29,205 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,205 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:29,205 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,206 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:29,206 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:29,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:29,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:29,206 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:29,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:29,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:29,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:29,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:29,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:29,208 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:29,208 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:29,208 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:29,208 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:29,208 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:29,209 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,209 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,209 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:29,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:29,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:29,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,210 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:29,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:29,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:29,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:29,211 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:29,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,213 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,213 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:29,213 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:29,213 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:29,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:29,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:29,213 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:29,213 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,213 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:29,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:29,213 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:29,213 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,213 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,213 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:29,213 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:29,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:29,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:29,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:29,226 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1128 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/90d9c010a0d943d8abeb27a95fbce5c2 2023-03-20 00:56:29,243 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1128 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/dc72064e26c24519b62ca55fd64604b2 2023-03-20 00:56:29,261 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1128 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/66b123c5a0c54a6b9b429d3b9b90aad6 2023-03-20 00:56:29,266 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/90d9c010a0d943d8abeb27a95fbce5c2 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/90d9c010a0d943d8abeb27a95fbce5c2 2023-03-20 00:56:29,270 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/90d9c010a0d943d8abeb27a95fbce5c2, entries=300, sequenceid=1128, filesize=19.0 K 2023-03-20 00:56:29,274 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/dc72064e26c24519b62ca55fd64604b2 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/dc72064e26c24519b62ca55fd64604b2 2023-03-20 00:56:29,280 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/dc72064e26c24519b62ca55fd64604b2, entries=150, sequenceid=1128, filesize=11.9 K 2023-03-20 00:56:29,280 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/66b123c5a0c54a6b9b429d3b9b90aad6 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/66b123c5a0c54a6b9b429d3b9b90aad6 2023-03-20 00:56:29,284 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/66b123c5a0c54a6b9b429d3b9b90aad6, entries=150, sequenceid=1128, filesize=11.9 K 2023-03-20 00:56:29,285 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~403.97 KB/413664, currentSize=40.25 KB/41220 for 51e645e88856df580ad0de225d47b16a in 478ms, sequenceid=1128, compaction requested=false 2023-03-20 00:56:29,286 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:29,313 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:29,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:29,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:29,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:29,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:29,334 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:29,334 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:29,334 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:29,334 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:29,334 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,334 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:29,335 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:29,335 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,335 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,335 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:29,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:29,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:29,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:29,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:29,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:29,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:29,351 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:29,351 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:29,352 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:29,352 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:29,352 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:29,352 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:29,352 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:29,352 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:29,352 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:29,352 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51e645e88856df580ad0de225d47b16a 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-20 00:56:29,352 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=A 2023-03-20 00:56:29,352 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:29,352 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=B 2023-03-20 00:56:29,352 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:29,352 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=C 2023-03-20 00:56:29,352 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:29,374 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=1137 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/c8ae9be0779c45de968ed2e4feef9081 2023-03-20 00:56:29,398 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=1137 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/803ade874f1d40c8bfd36d35f0246345 2023-03-20 00:56:29,414 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=1137 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/97a5beb404104b80a193fc2dd25b5409 2023-03-20 00:56:29,418 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/c8ae9be0779c45de968ed2e4feef9081 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/c8ae9be0779c45de968ed2e4feef9081 2023-03-20 00:56:29,421 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/c8ae9be0779c45de968ed2e4feef9081, entries=150, sequenceid=1137, filesize=11.9 K 2023-03-20 00:56:29,422 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/803ade874f1d40c8bfd36d35f0246345 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/803ade874f1d40c8bfd36d35f0246345 2023-03-20 00:56:29,426 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/803ade874f1d40c8bfd36d35f0246345, entries=150, sequenceid=1137, filesize=11.9 K 2023-03-20 00:56:29,427 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/97a5beb404104b80a193fc2dd25b5409 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/97a5beb404104b80a193fc2dd25b5409 2023-03-20 00:56:29,431 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/97a5beb404104b80a193fc2dd25b5409, entries=150, sequenceid=1137, filesize=11.9 K 2023-03-20 00:56:29,431 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for 51e645e88856df580ad0de225d47b16a in 79ms, sequenceid=1137, compaction requested=true 2023-03-20 00:56:29,431 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:29,431 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:29,431 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:29,431 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:29,431 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,431 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:29,432 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:29,432 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,432 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:29,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:29,433 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,433 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:29,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:29,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:29,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:29,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:29,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2730e6f6[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:29,433 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:29,434 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,434 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:29,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:29,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:29,434 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,434 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:29,435 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,435 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:29,435 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:29,435 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:29,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:29,437 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:29,437 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:29,437 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:29,437 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:29,437 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:29,438 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,438 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,438 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:29,438 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,440 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:29,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:29,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,441 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:29,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:29,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:29,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,442 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:29,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:29,442 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:29,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:29,442 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:29,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:29,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:29,457 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51e645e88856df580ad0de225d47b16a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-20 00:56:29,458 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=A 2023-03-20 00:56:29,458 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:29,458 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=B 2023-03-20 00:56:29,458 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:29,458 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=C 2023-03-20 00:56:29,458 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:29,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:29,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273849472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679273789469, max timestamp=1679273789474Num uniques 859; 2023-03-20 00:56:29,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 00:56:29,483 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:29,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273849480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,483 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:29,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273849480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,483 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:29,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273849481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,486 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:29,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273849483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,496 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1148 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/1b2b0a7eca38407ea9d4f01b9c30ff87 2023-03-20 00:56:29,542 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:29,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:29,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:29,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:29,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:29,554 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:29,554 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:29,554 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:29,554 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:29,555 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,555 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:29,571 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:29,571 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:29,571 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,572 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:29,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:29,572 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:29,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:29,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:29,578 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:29,578 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:29,578 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:29,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:29,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273849576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:29,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273849584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,586 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:29,586 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:29,586 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:29,586 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:29,586 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:29,586 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:29,586 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:29,586 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:29,586 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,586 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:29,586 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:29,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:29,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273849590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:29,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273849591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:29,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273849593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,708 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,708 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:29,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:29,708 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,708 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:29,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:29,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:29,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:29,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:29,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31b6e8ae[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:29,710 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:29,710 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,711 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,711 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:29,711 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,711 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:29,711 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:29,711 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:29,712 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:29,712 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:29,712 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:29,712 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:29,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:29,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:29,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:29,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:29,714 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:29,714 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:29,714 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:29,714 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:29,715 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:29,715 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,716 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,716 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:29,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:29,716 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,717 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:29,718 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,718 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:29,718 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:29,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:29,718 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,718 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:29,718 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:29,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:29,719 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:29,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:29,719 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:29,719 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:29,719 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,719 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:29,719 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:29,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:29,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:29,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273849781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,787 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:29,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273849785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:29,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273849795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:29,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:29,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273849795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273849796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,819 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:29,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:29,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:29,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:29,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:29,823 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:29,823 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:29,823 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:29,823 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:29,824 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,824 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:29,825 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:29,825 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:29,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:29,825 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,825 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:29,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:29,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:29,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:29,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:29,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:29,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:29,830 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:29,830 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:29,830 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:29,832 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:29,832 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:29,832 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:29,832 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:29,832 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:29,832 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:29,832 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:29,832 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:29,832 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,832 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:29,832 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:29,833 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:29,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:29,833 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:29,833 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,833 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:29,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:29,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:29,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:29,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ff48827[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:29,834 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:29,834 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,835 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,835 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:29,835 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:29,835 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,835 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:29,835 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:29,836 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,836 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:29,836 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:29,836 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:29,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:29,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:29,837 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:29,838 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:29,838 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:29,838 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:29,838 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:29,838 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,838 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:29,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:29,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,838 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:29,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:29,839 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:29,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:29,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:29,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,840 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:29,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,841 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,841 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:29,841 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:29,841 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,841 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,841 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:29,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:29,841 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:29,841 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,841 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:29,841 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:29,842 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:29,842 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:29,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:29,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:29,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:29,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:29,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:29,912 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1148 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/38bacc889e904b59817c75977d67c995 2023-03-20 00:56:29,942 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:29,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:29,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:29,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:29,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:29,945 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:29,945 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:29,945 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:29,945 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:29,945 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,945 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:29,946 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:29,946 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:29,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:29,946 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,946 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:29,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:29,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:29,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:29,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:29,950 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:29,950 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:29,950 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:29,950 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:29,950 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:29,950 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:29,950 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:29,950 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:29,950 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:29,950 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:29,951 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:29,951 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:29,951 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:29,951 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,951 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:29,951 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:29,952 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,952 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:29,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:29,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:29,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:29,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,953 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,953 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:29,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:29,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:29,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a1591b0[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:29,954 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:29,954 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,955 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,955 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:29,955 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:29,955 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:29,955 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,955 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:29,956 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,956 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:29,956 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:29,956 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:29,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:29,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:29,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:29,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:29,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:29,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:29,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:29,960 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:29,960 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:29,960 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:29,960 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:29,960 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:29,961 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,961 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:29,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,961 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:29,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:29,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:29,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:29,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:29,962 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:29,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,964 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:29,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,966 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,966 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:29,966 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:29,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:29,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:29,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:29,966 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:29,966 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:29,966 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:29,966 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:29,966 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,966 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:29,966 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:29,966 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:29,966 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:29,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:29,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:29,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:30,066 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:30,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:30,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:30,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:30,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:30,071 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:30,071 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:30,071 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:30,071 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:30,078 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:30,078 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:30,079 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:30,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:30,079 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:30,079 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,079 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:30,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:30,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:30,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:30,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:30,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:30,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:30,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:30,080 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:30,080 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:30,080 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:30,082 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:30,082 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:30,082 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:30,082 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:30,082 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:30,082 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:30,082 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:30,082 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:30,082 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,082 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:30,082 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:30,083 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:30,083 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:30,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:30,083 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:30,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:30,083 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:30,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:30,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:30,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:30,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:30,085 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:30,085 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:30,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a32d752[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:30,085 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:30,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:30,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:30,086 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:30,086 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:30,086 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,086 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:30,086 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:30,087 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,087 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:30,087 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:30,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:30,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:30,087 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:30,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:30,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:30,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273850085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:30,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:30,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:30,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:30,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:30,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:30,089 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:30,089 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:30,089 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:30,089 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:30,089 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:30,090 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:30,090 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:30,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:30,090 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:30,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:30,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:30,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:30,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:30,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:30,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273850089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:30,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:30,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:30,091 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:30,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:30,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:30,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:30,092 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:30,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:30,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,094 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,094 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:30,094 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:30,094 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:30,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:30,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:30,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:30,094 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:30,094 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:30,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:30,094 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,094 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:30,094 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:30,094 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:30,095 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:30,095 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:30,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:30,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:30,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:30,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273850098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:30,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273850099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:30,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273850100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,195 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:30,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:30,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:30,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:30,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:30,214 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:30,214 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:30,214 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:30,214 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:30,215 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:30,215 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:30,215 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,215 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:30,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:30,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:30,216 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,216 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:30,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:30,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:30,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:30,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:30,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:30,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:30,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:30,222 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:30,222 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:30,222 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:30,238 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:30,238 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:30,238 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:30,238 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:30,238 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:30,238 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:30,238 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:30,238 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:30,238 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,238 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:30,238 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:30,243 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:30,243 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:30,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:30,243 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:30,243 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:30,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:30,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:30,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:30,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:30,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:30,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c47af22[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:30,245 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:30,246 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:30,246 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:30,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:30,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:30,246 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:30,246 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:30,246 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:30,246 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,247 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:30,247 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:30,247 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:30,247 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:30,248 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:30,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:30,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:30,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:30,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:30,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:30,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:30,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:30,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:30,250 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:30,250 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:30,250 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:30,250 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:30,250 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:30,252 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:30,252 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:30,252 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:30,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:30,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:30,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:30,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:30,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:30,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:30,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:30,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:30,252 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:30,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:30,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:30,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:30,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:30,254 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:30,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,256 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:30,257 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:30,257 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,257 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:30,257 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:30,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:30,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:30,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:30,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:30,257 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:30,257 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:30,257 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:30,257 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,257 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:30,257 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:30,257 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:30,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:30,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:30,330 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1148 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/96b800f77f414819ad4b52c43de89c94 2023-03-20 00:56:30,336 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/1b2b0a7eca38407ea9d4f01b9c30ff87 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/1b2b0a7eca38407ea9d4f01b9c30ff87 2023-03-20 00:56:30,341 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/1b2b0a7eca38407ea9d4f01b9c30ff87, entries=150, sequenceid=1148, filesize=11.9 K 2023-03-20 00:56:30,343 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/38bacc889e904b59817c75977d67c995 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/38bacc889e904b59817c75977d67c995 2023-03-20 00:56:30,347 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/38bacc889e904b59817c75977d67c995, entries=150, sequenceid=1148, filesize=11.9 K 2023-03-20 00:56:30,348 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/96b800f77f414819ad4b52c43de89c94 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/96b800f77f414819ad4b52c43de89c94 2023-03-20 00:56:30,352 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/96b800f77f414819ad4b52c43de89c94, entries=150, sequenceid=1148, filesize=11.9 K 2023-03-20 00:56:30,353 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=123.00 KB/125950 for 51e645e88856df580ad0de225d47b16a in 896ms, sequenceid=1148, compaction requested=true 2023-03-20 00:56:30,353 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:30,353 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:30,353 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 00:56:30,353 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 00:56:30,353 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:30,354 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:30,357 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:30,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:30,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:30,360 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51078 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 00:56:30,360 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 51e645e88856df580ad0de225d47b16a/A is initiating minor compaction (all files) 2023-03-20 00:56:30,361 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e645e88856df580ad0de225d47b16a/A in TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:30,361 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/b96464e5dfc8454190814d7d4215e006, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/90d9c010a0d943d8abeb27a95fbce5c2, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/c8ae9be0779c45de968ed2e4feef9081, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/1b2b0a7eca38407ea9d4f01b9c30ff87] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp, totalSize=49.9 K 2023-03-20 00:56:30,361 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting b96464e5dfc8454190814d7d4215e006, keycount=50, bloomtype=ROW, size=7.0 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1679273787458 2023-03-20 00:56:30,362 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 90d9c010a0d943d8abeb27a95fbce5c2, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=1128, earliestPutTs=1679273788769 2023-03-20 00:56:30,362 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 43758 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 00:56:30,362 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 51e645e88856df580ad0de225d47b16a/B is initiating minor compaction (all files) 2023-03-20 00:56:30,362 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e645e88856df580ad0de225d47b16a/B in TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:30,362 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting c8ae9be0779c45de968ed2e4feef9081, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1137, earliestPutTs=1679273788814 2023-03-20 00:56:30,362 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/ab9d24979c6644e09cceec5bf045d38e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/dc72064e26c24519b62ca55fd64604b2, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/803ade874f1d40c8bfd36d35f0246345, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/38bacc889e904b59817c75977d67c995] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp, totalSize=42.7 K 2023-03-20 00:56:30,362 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 1b2b0a7eca38407ea9d4f01b9c30ff87, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1148, earliestPutTs=1679273789453 2023-03-20 00:56:30,364 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting ab9d24979c6644e09cceec5bf045d38e, keycount=50, bloomtype=ROW, size=7.0 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1679273787458 2023-03-20 00:56:30,364 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting dc72064e26c24519b62ca55fd64604b2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1128, earliestPutTs=1679273788769 2023-03-20 00:56:30,365 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 803ade874f1d40c8bfd36d35f0246345, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1137, earliestPutTs=1679273788814 2023-03-20 00:56:30,365 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 38bacc889e904b59817c75977d67c995, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1148, earliestPutTs=1679273789453 2023-03-20 00:56:30,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:30,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:30,366 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:30,366 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:30,366 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:30,366 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:30,366 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:30,367 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:30,367 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:30,367 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:30,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:30,368 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,368 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:30,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:30,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:30,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:30,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:30,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:30,374 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:30,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:30,374 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:30,374 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:30,374 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:30,382 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:30,382 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:30,382 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:30,382 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:30,384 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51e645e88856df580ad0de225d47b16a 3/3 column families, dataSize=123.00 KB heapSize=361.72 KB 2023-03-20 00:56:30,384 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=A 2023-03-20 00:56:30,385 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:56:30,385 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=B 2023-03-20 00:56:30,385 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 51e645e88856df580ad0de225d47b16a#B#compaction#189 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:30,386 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:56:30,386 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=C 2023-03-20 00:56:30,386 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:56:30,389 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51e645e88856df580ad0de225d47b16a#A#compaction#190 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:30,435 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/45b5dd34403e43a684e66863b8cafd28 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/45b5dd34403e43a684e66863b8cafd28 2023-03-20 00:56:30,439 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 51e645e88856df580ad0de225d47b16a/A of 51e645e88856df580ad0de225d47b16a into 45b5dd34403e43a684e66863b8cafd28(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:30,439 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:30,439 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a., storeName=51e645e88856df580ad0de225d47b16a/A, priority=12, startTime=1679273790353; duration=0sec 2023-03-20 00:56:30,439 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:30,439 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 00:56:30,441 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 43758 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 00:56:30,441 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 51e645e88856df580ad0de225d47b16a/C is initiating minor compaction (all files) 2023-03-20 00:56:30,441 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e645e88856df580ad0de225d47b16a/C in TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:30,441 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/da23df92d8404f5d955401c04ae9374d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/66b123c5a0c54a6b9b429d3b9b90aad6, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/97a5beb404104b80a193fc2dd25b5409, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/96b800f77f414819ad4b52c43de89c94] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp, totalSize=42.7 K 2023-03-20 00:56:30,441 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting da23df92d8404f5d955401c04ae9374d, keycount=50, bloomtype=ROW, size=7.0 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1679273787458 2023-03-20 00:56:30,441 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 66b123c5a0c54a6b9b429d3b9b90aad6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1128, earliestPutTs=1679273788769 2023-03-20 00:56:30,442 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 97a5beb404104b80a193fc2dd25b5409, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1137, earliestPutTs=1679273788814 2023-03-20 00:56:30,442 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 96b800f77f414819ad4b52c43de89c94, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1148, earliestPutTs=1679273789453 2023-03-20 00:56:30,448 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51e645e88856df580ad0de225d47b16a#C#compaction#192 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:30,502 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-20 00:56:30,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:30,593 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:30,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:30,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273850607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:30,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273850609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:30,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273850610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:30,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273850610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:30,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273850610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:30,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273850713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:30,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273850714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:30,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273850715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:30,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273850715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:30,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273850715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,812 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/75efe15dbbd145028b28980d9ee985e7 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/75efe15dbbd145028b28980d9ee985e7 2023-03-20 00:56:30,818 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 51e645e88856df580ad0de225d47b16a/B of 51e645e88856df580ad0de225d47b16a into 75efe15dbbd145028b28980d9ee985e7(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:30,818 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:30,818 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a., storeName=51e645e88856df580ad0de225d47b16a/B, priority=12, startTime=1679273790353; duration=0sec 2023-03-20 00:56:30,818 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:30,830 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1174 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/409496aa6b024011963b35c34266a476 2023-03-20 00:56:30,906 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/2c0ca12c3f134637bc61876ce25198c9 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/2c0ca12c3f134637bc61876ce25198c9 2023-03-20 00:56:30,910 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 51e645e88856df580ad0de225d47b16a/C of 51e645e88856df580ad0de225d47b16a into 2c0ca12c3f134637bc61876ce25198c9(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:30,910 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:30,910 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a., storeName=51e645e88856df580ad0de225d47b16a/C, priority=12, startTime=1679273790354; duration=0sec 2023-03-20 00:56:30,910 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:30,917 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:30,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273850916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:30,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273850916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:30,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273850917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:30,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273850918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:30,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:30,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273850918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:31,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273851218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:31,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273851219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:31,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273851220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:31,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273851220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:31,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273851222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,252 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=1174 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/e919b624d6534be39bf82f585fd82f71 2023-03-20 00:56:31,664 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=1174 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/3c7aeea2316042e3b529cf26896e27b8 2023-03-20 00:56:31,670 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/409496aa6b024011963b35c34266a476 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/409496aa6b024011963b35c34266a476 2023-03-20 00:56:31,695 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/409496aa6b024011963b35c34266a476, entries=150, sequenceid=1174, filesize=11.9 K 2023-03-20 00:56:31,696 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/e919b624d6534be39bf82f585fd82f71 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/e919b624d6534be39bf82f585fd82f71 2023-03-20 00:56:31,708 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/e919b624d6534be39bf82f585fd82f71, entries=150, sequenceid=1174, filesize=11.9 K 2023-03-20 00:56:31,709 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/3c7aeea2316042e3b529cf26896e27b8 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/3c7aeea2316042e3b529cf26896e27b8 2023-03-20 00:56:31,724 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/3c7aeea2316042e3b529cf26896e27b8, entries=150, sequenceid=1174, filesize=11.9 K 2023-03-20 00:56:31,724 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:31,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273851723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,725 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~123.00 KB/125950, heapSize ~361.67 KB/370352, currentSize=60.38 KB/61830 for 51e645e88856df580ad0de225d47b16a in 1342ms, sequenceid=1174, compaction requested=false 2023-03-20 00:56:31,725 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:31,725 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:31,725 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:31,725 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:31,725 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,726 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:31,726 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:31,727 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,727 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:31,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:31,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:31,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:31,728 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51e645e88856df580ad0de225d47b16a 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-20 00:56:31,729 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=A 2023-03-20 00:56:31,729 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:31,729 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=B 2023-03-20 00:56:31,729 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:31,729 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=C 2023-03-20 00:56:31,729 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:31,731 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:31,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:31,731 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:31,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:31,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:31,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:31,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:31,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1779a93d[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:31,738 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:31,738 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:31,739 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:31,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:31,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:31,740 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:31,740 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:31,740 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:31,740 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,740 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:31,740 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:31,740 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,741 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:31,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:31,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:31,741 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:31,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:31,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:31,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:31,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:31,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:31,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:31,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:31,745 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:31,745 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:31,745 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:31,745 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:31,745 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:31,746 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:31,746 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:31,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:31,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:31,746 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:31,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:31,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:31,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:31,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:31,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:31,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:31,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:31,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:31,747 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:31,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:31,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:31,749 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:31,750 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273851744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:31,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:31,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273851745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,752 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:31,752 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:31,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:31,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:31,753 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,753 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:31,753 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:31,753 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,753 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:31,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:31,753 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:31,753 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:31,753 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:31,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:31,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:31,753 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:31,753 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:31,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:31,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273851750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,761 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:31,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273851750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:31,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273851851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,852 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:31,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273851851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,853 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:31,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:31,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:31,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:31,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:31,858 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:31,858 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:31,858 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:31,858 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:31,859 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:31,859 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:31,860 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:31,860 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:31,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:31,861 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,861 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:31,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:31,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273851862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:31,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273851863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:31,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:31,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:31,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:31,871 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:31,872 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:31,872 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:31,872 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:31,872 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:31,872 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:31,874 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:31,874 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:31,874 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:31,874 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:31,874 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:31,874 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:31,874 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:31,874 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:31,874 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,874 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:31,874 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:31,875 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:31,875 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:31,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:31,876 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:31,876 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:31,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:31,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:31,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:31,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:31,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:31,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f73273[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:31,877 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:31,877 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:31,878 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:31,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:31,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:31,878 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:31,878 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:31,878 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:31,878 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,878 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:31,879 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:31,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:31,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:31,880 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:31,880 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:31,880 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:31,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:31,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:31,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:31,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:31,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:31,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:31,886 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:31,886 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:31,886 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:31,886 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:31,886 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:31,891 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:31,892 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:31,892 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:31,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:31,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:31,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:31,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:31,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:31,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:31,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:31,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:31,893 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:31,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:31,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:31,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:31,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:31,895 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:31,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,900 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:31,900 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:31,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:31,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:31,900 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,900 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:31,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:31,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:31,900 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:31,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:31,900 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:31,900 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:31,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:31,900 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:31,901 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:31,901 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:31,901 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:31,901 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,001 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:32,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:32,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:32,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:32,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:32,003 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:32,004 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:32,004 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:32,004 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:32,004 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,004 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:32,005 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:32,005 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:32,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:32,005 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,005 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:32,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:32,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:32,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:32,006 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:32,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:32,007 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:32,007 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:32,007 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:32,007 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:32,007 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:32,007 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:32,007 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:32,007 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:32,007 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:32,007 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:32,007 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:32,007 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:32,007 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,007 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:32,007 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:32,008 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,008 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:32,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:32,008 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,008 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:32,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:32,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:32,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:32,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:32,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:32,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d73ed58[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:32,010 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:32,010 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,010 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,010 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:32,010 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:32,010 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:32,010 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,011 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:32,012 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:32,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:32,012 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:32,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:32,012 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:32,013 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:32,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:32,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:32,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:32,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:32,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:32,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:32,014 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:32,014 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:32,014 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:32,014 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:32,014 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:32,015 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:32,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:32,015 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,015 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:32,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:32,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:32,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:32,016 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:32,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:32,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:32,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,017 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:32,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:32,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,019 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,019 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:32,019 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:32,019 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,019 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,019 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,019 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:32,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:32,019 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:32,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:32,020 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:32,020 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:32,020 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:32,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:32,020 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:32,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:32,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:32,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:32,053 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:32,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273852053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,054 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:32,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273852053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:32,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273852067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:32,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273852067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,120 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:32,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:32,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:32,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:32,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:32,123 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:32,123 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:32,123 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:32,123 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:32,123 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,123 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:32,124 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,124 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:32,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:32,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:32,124 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,124 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:32,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:32,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:32,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:32,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:32,125 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:32,125 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:32,125 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:32,125 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:32,125 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:32,125 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:32,125 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:32,125 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:32,125 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:32,125 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:32,125 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:32,125 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:32,125 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:32,125 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,125 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:32,125 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:32,126 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,126 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:32,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:32,126 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,126 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:32,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:32,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:32,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:32,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:32,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:32,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c6c7d6b[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:32,128 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:32,128 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,128 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,128 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:32,128 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:32,128 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:32,128 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,128 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:32,129 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,129 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:32,129 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:32,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,129 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:32,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:32,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:32,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:32,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:32,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:32,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:32,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:32,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:32,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:32,131 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:32,131 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:32,131 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:32,131 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:32,131 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:32,132 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,132 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:32,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:32,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,132 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:32,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:32,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:32,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:32,132 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:32,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:32,133 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:32,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:32,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:32,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,134 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,134 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:32,135 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:32,135 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:32,135 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,135 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,135 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,135 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:32,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:32,135 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:32,135 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:32,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:32,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:32,135 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:32,135 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:32,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:32,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:32,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:32,166 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1191 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/792dca22532f49fd81bacc7a4b895bc6 2023-03-20 00:56:32,235 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:32,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:32,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:32,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:32,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:32,238 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:32,238 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:32,238 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:32,239 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:32,239 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,239 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:32,240 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,240 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,240 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:32,242 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:32,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:32,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:32,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:32,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:32,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:32,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:32,250 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:32,250 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:32,250 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:32,250 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:32,250 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:32,258 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:32,258 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:32,258 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:32,258 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:32,258 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:32,258 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:32,258 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:32,258 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:32,258 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,258 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:32,259 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:32,260 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,260 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:32,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:32,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:32,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:32,261 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,261 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:32,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:32,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:32,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:32,262 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:32,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7606199a[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:32,262 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,263 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,263 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:32,263 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:32,263 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:32,264 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,264 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:32,264 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:32,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:32,264 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:32,264 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:32,264 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:32,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:32,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:32,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:32,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:32,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:32,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:32,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:32,272 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:32,272 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:32,272 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:32,272 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:32,272 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:32,275 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,275 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,275 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:32,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:32,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:32,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:32,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:32,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:32,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,277 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:32,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:32,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:32,278 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:32,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:32,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,280 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:32,280 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:32,280 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:32,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:32,280 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:32,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:32,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:32,280 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:32,281 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:32,281 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:32,281 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:32,281 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,281 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,281 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,281 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:32,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:32,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:32,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:32,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:32,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273852355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273852355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:32,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273852371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:32,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273852372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,381 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:32,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:32,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:32,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:32,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:32,386 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:32,386 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:32,386 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:32,386 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:32,386 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,386 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:32,387 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:32,387 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:32,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:32,387 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,387 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:32,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:32,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:32,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:32,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:32,389 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:32,389 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:32,389 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:32,389 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:32,389 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:32,389 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:32,389 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:32,389 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:32,389 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:32,389 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:32,389 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:32,389 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:32,389 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:32,390 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,390 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:32,390 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:32,391 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,391 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:32,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:32,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:32,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:32,392 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,392 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:32,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:32,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:32,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:32,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e5ac983[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:32,394 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:32,394 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,395 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,395 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:32,395 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:32,395 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:32,395 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,395 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:32,396 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:32,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:32,396 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:32,396 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:32,396 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:32,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:32,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:32,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:32,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:32,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:32,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:32,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:32,400 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:32,400 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:32,400 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:32,400 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:32,400 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:32,409 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,409 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:32,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:32,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,409 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:32,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:32,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:32,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:32,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,410 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:32,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:32,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:32,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:32,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:32,413 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,416 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,416 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:32,417 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:32,417 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:32,417 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:32,417 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,417 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:32,417 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:32,417 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,417 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:32,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:32,417 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:32,417 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:32,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:32,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:32,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:32,517 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:32,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:32,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:32,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:32,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:32,530 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:32,530 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:32,530 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:32,530 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:32,531 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,531 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:32,532 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,533 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:32,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:32,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:32,533 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,533 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:32,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:32,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:32,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:32,535 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:32,545 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:32,545 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:32,545 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:32,545 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:32,545 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:32,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:32,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:32,550 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:32,550 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:32,550 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:32,550 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:32,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:32,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:32,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:32,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:32,551 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:32,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:32,552 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:32,552 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,552 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:32,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:32,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:32,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:32,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:32,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c51e43d[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:32,556 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:32,556 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,558 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,558 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,558 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:32,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,559 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:32,559 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:32,559 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:32,560 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:32,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:32,561 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:32,561 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:32,561 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:32,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:32,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:32,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:32,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:32,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:32,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:32,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:32,571 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:32,571 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:32,571 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:32,572 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:32,572 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:32,572 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,572 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:32,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,573 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:32,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:32,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:32,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:32,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:32,573 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:32,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:32,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:32,576 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:32,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:32,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,577 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,577 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:32,577 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:32,577 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:32,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:32,577 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:32,577 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,578 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:32,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:32,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:32,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:32,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:32,578 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:32,578 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:32,578 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:32,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:32,578 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:32,578 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,592 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1191 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/321bb3c6b2594ba5aa99a04c8abfe62c 2023-03-20 00:56:32,658 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1191 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/cb6eea3068404a5880f3d26992650dc8 2023-03-20 00:56:32,663 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/792dca22532f49fd81bacc7a4b895bc6 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/792dca22532f49fd81bacc7a4b895bc6 2023-03-20 00:56:32,669 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/792dca22532f49fd81bacc7a4b895bc6, entries=150, sequenceid=1191, filesize=11.9 K 2023-03-20 00:56:32,670 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/321bb3c6b2594ba5aa99a04c8abfe62c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/321bb3c6b2594ba5aa99a04c8abfe62c 2023-03-20 00:56:32,675 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/321bb3c6b2594ba5aa99a04c8abfe62c, entries=150, sequenceid=1191, filesize=11.9 K 2023-03-20 00:56:32,678 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/cb6eea3068404a5880f3d26992650dc8 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/cb6eea3068404a5880f3d26992650dc8 2023-03-20 00:56:32,678 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:32,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:32,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:32,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:32,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:32,681 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:32,682 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:32,683 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:32,683 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:32,689 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,690 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:32,691 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:32,691 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:32,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:32,692 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/cb6eea3068404a5880f3d26992650dc8, entries=150, sequenceid=1191, filesize=11.9 K 2023-03-20 00:56:32,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:32,693 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:32,693 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 51e645e88856df580ad0de225d47b16a in 965ms, sequenceid=1191, compaction requested=true 2023-03-20 00:56:32,693 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:32,693 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:32,693 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:56:32,693 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:56:32,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:32,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:32,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:32,693 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:32,693 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:32,694 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:32,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:32,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:32,695 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:32,695 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:32,695 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:32,697 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:32,697 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:32,697 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:32,697 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:32,697 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51e645e88856df580ad0de225d47b16a 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-20 00:56:32,697 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=A 2023-03-20 00:56:32,697 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:32,697 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=B 2023-03-20 00:56:32,698 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:32,698 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=C 2023-03-20 00:56:32,698 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:32,699 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36696 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:56:32,699 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 51e645e88856df580ad0de225d47b16a/A is initiating minor compaction (all files) 2023-03-20 00:56:32,699 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e645e88856df580ad0de225d47b16a/A in TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:32,699 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/45b5dd34403e43a684e66863b8cafd28, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/409496aa6b024011963b35c34266a476, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/792dca22532f49fd81bacc7a4b895bc6] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp, totalSize=35.8 K 2023-03-20 00:56:32,700 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36696 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:56:32,700 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 51e645e88856df580ad0de225d47b16a/B is initiating minor compaction (all files) 2023-03-20 00:56:32,700 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e645e88856df580ad0de225d47b16a/B in TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:32,700 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 45b5dd34403e43a684e66863b8cafd28, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=1148, earliestPutTs=1679273789453 2023-03-20 00:56:32,700 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/75efe15dbbd145028b28980d9ee985e7, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/e919b624d6534be39bf82f585fd82f71, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/321bb3c6b2594ba5aa99a04c8abfe62c] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp, totalSize=35.8 K 2023-03-20 00:56:32,702 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 75efe15dbbd145028b28980d9ee985e7, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=1148, earliestPutTs=1679273789453 2023-03-20 00:56:32,702 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting e919b624d6534be39bf82f585fd82f71, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1174, earliestPutTs=1679273789474 2023-03-20 00:56:32,703 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 409496aa6b024011963b35c34266a476, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1174, earliestPutTs=1679273789474 2023-03-20 00:56:32,703 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 321bb3c6b2594ba5aa99a04c8abfe62c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1191, earliestPutTs=1679273790608 2023-03-20 00:56:32,703 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 792dca22532f49fd81bacc7a4b895bc6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1191, earliestPutTs=1679273790608 2023-03-20 00:56:32,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:32,731 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:32,733 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51e645e88856df580ad0de225d47b16a#A#compaction#199 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:32,744 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 51e645e88856df580ad0de225d47b16a#B#compaction#200 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:32,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:32,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273852778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,782 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/bd64cb659a56423ab60743556fc573da as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/bd64cb659a56423ab60743556fc573da 2023-03-20 00:56:32,788 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 51e645e88856df580ad0de225d47b16a/B of 51e645e88856df580ad0de225d47b16a into bd64cb659a56423ab60743556fc573da(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:32,788 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:32,788 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a., storeName=51e645e88856df580ad0de225d47b16a/B, priority=13, startTime=1679273792693; duration=0sec 2023-03-20 00:56:32,789 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:32,789 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:56:32,790 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36696 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:56:32,790 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 51e645e88856df580ad0de225d47b16a/C is initiating minor compaction (all files) 2023-03-20 00:56:32,790 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e645e88856df580ad0de225d47b16a/C in TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:32,790 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/2c0ca12c3f134637bc61876ce25198c9, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/3c7aeea2316042e3b529cf26896e27b8, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/cb6eea3068404a5880f3d26992650dc8] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp, totalSize=35.8 K 2023-03-20 00:56:32,791 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 2c0ca12c3f134637bc61876ce25198c9, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=1148, earliestPutTs=1679273789453 2023-03-20 00:56:32,792 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 3c7aeea2316042e3b529cf26896e27b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1174, earliestPutTs=1679273789474 2023-03-20 00:56:32,792 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting cb6eea3068404a5880f3d26992650dc8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1191, earliestPutTs=1679273790608 2023-03-20 00:56:32,801 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 51e645e88856df580ad0de225d47b16a#C#compaction#201 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:32,842 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/d43d7fe6286a4996a12b0566a6534473 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/d43d7fe6286a4996a12b0566a6534473 2023-03-20 00:56:32,849 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 51e645e88856df580ad0de225d47b16a/C of 51e645e88856df580ad0de225d47b16a into d43d7fe6286a4996a12b0566a6534473(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:32,849 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:32,849 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a., storeName=51e645e88856df580ad0de225d47b16a/C, priority=13, startTime=1679273792694; duration=0sec 2023-03-20 00:56:32,849 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:32,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:32,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273852858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:32,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273852858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:32,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273852873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:32,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273852875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:32,881 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:32,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273852880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:33,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:33,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273853083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:33,172 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1210 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/f8373e59a336466ea90c2c02c076b395 2023-03-20 00:56:33,180 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/c0939baa4fa64dd3b5e720389e45128c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/c0939baa4fa64dd3b5e720389e45128c 2023-03-20 00:56:33,185 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 51e645e88856df580ad0de225d47b16a/A of 51e645e88856df580ad0de225d47b16a into c0939baa4fa64dd3b5e720389e45128c(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:33,185 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:33,185 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a., storeName=51e645e88856df580ad0de225d47b16a/A, priority=13, startTime=1679273792693; duration=0sec 2023-03-20 00:56:33,185 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:33,195 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1210 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/fe22c3bd3d864e08b4a181bbb9ef43d3 2023-03-20 00:56:33,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:33,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273853386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:33,604 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1210 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/e1144cb1ff44420095aad397b51b89ae 2023-03-20 00:56:33,608 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/f8373e59a336466ea90c2c02c076b395 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/f8373e59a336466ea90c2c02c076b395 2023-03-20 00:56:33,611 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/f8373e59a336466ea90c2c02c076b395, entries=150, sequenceid=1210, filesize=11.9 K 2023-03-20 00:56:33,612 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/fe22c3bd3d864e08b4a181bbb9ef43d3 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/fe22c3bd3d864e08b4a181bbb9ef43d3 2023-03-20 00:56:33,616 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/fe22c3bd3d864e08b4a181bbb9ef43d3, entries=150, sequenceid=1210, filesize=11.9 K 2023-03-20 00:56:33,617 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/e1144cb1ff44420095aad397b51b89ae as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/e1144cb1ff44420095aad397b51b89ae 2023-03-20 00:56:33,621 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/e1144cb1ff44420095aad397b51b89ae, entries=150, sequenceid=1210, filesize=11.9 K 2023-03-20 00:56:33,623 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 51e645e88856df580ad0de225d47b16a in 926ms, sequenceid=1210, compaction requested=false 2023-03-20 00:56:33,624 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:33,624 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:33,624 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:33,624 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:33,624 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:33,624 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:33,624 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:33,625 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:33,625 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:33,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:33,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:33,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:33,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:33,626 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:33,626 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:33,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:33,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:33,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:33,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:33,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:33,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13195df9[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:33,627 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:33,627 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:33,627 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:33,627 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:33,627 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:33,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:33,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:33,628 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:33,628 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:33,628 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:33,628 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:33,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:33,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:33,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:33,628 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:33,628 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:33,629 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:33,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:33,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:33,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:33,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:33,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:33,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:33,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:33,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:33,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:33,631 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:33,631 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:33,631 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:33,631 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:33,631 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:33,632 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:33,632 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:33,632 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:33,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:33,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:33,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:33,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:33,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:33,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:33,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:33,633 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:33,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:33,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:33,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:33,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:33,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:33,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:33,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:33,635 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:33,636 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:33,636 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:33,636 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:33,636 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:33,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:33,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:33,637 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:33,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:33,637 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:33,637 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:33,637 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:33,637 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:33,637 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:33,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:33,637 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:33,637 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:33,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:33,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:33,737 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:33,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:33,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:33,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:33,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:33,741 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:33,741 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:33,741 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:33,741 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:33,741 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:33,741 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:33,743 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:33,743 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:33,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:33,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:33,743 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:33,743 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:33,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:33,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:33,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:33,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:33,744 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:33,744 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:33,744 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:33,744 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:33,744 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:33,744 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:33,744 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:33,744 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:33,744 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:33,744 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:33,745 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51e645e88856df580ad0de225d47b16a 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-20 00:56:33,745 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=A 2023-03-20 00:56:33,745 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:33,745 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=B 2023-03-20 00:56:33,745 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:33,745 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=C 2023-03-20 00:56:33,746 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:33,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:33,867 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:33,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:33,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273853892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:33,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:33,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273853892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:33,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:33,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273853896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:33,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:33,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273853896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:33,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:33,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273853897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:34,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273853998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,003 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:34,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273854001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:34,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273854001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:34,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273854002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,167 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1226 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/7e78a26f5bde488bac895b60289e8b12 2023-03-20 00:56:34,187 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1226 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/079f4bb06b2a4e69a32dbd9b19253ab2 2023-03-20 00:56:34,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:34,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273854201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:34,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273854205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:34,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273854205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:34,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273854211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,223 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1226 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/8831a9714ff44778a8429cbd143177b2 2023-03-20 00:56:34,229 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/7e78a26f5bde488bac895b60289e8b12 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/7e78a26f5bde488bac895b60289e8b12 2023-03-20 00:56:34,235 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/7e78a26f5bde488bac895b60289e8b12, entries=150, sequenceid=1226, filesize=11.9 K 2023-03-20 00:56:34,235 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/079f4bb06b2a4e69a32dbd9b19253ab2 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/079f4bb06b2a4e69a32dbd9b19253ab2 2023-03-20 00:56:34,248 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/079f4bb06b2a4e69a32dbd9b19253ab2, entries=150, sequenceid=1226, filesize=11.9 K 2023-03-20 00:56:34,248 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/8831a9714ff44778a8429cbd143177b2 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/8831a9714ff44778a8429cbd143177b2 2023-03-20 00:56:34,273 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/8831a9714ff44778a8429cbd143177b2, entries=150, sequenceid=1226, filesize=11.9 K 2023-03-20 00:56:34,274 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 51e645e88856df580ad0de225d47b16a in 530ms, sequenceid=1226, compaction requested=true 2023-03-20 00:56:34,274 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:34,274 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:34,275 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:34,275 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:34,275 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,276 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:34,276 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:34,277 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,277 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:34,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:34,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:34,277 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:34,277 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:34,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:34,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:34,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:34,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:34,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:34,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55155cc9[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:34,279 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:34,279 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:34,280 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:34,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:34,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:34,280 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:34,280 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:34,280 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:34,280 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,281 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:34,281 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:34,281 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:34,281 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:34,281 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:34,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:34,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:34,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:34,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:34,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:34,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:34,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:34,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:34,288 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:34,288 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:34,288 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:34,288 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:34,288 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:34,292 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:34,292 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:34,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:34,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:34,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:34,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:34,292 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:34,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:34,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:34,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:34,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:34,293 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:34,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:34,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:34,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:34,294 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:34,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:34,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,297 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:34,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:34,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:34,298 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,298 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:34,298 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:34,298 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,298 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:34,298 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:34,298 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:34,298 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:34,298 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:34,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:34,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:34,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:34,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:34,298 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:34,298 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:34,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:34,399 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:34,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:34,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:34,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:34,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:34,406 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:34,406 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:34,406 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:34,406 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:34,406 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:34,406 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:34,407 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:34,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:34,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:34,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:34,407 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:34,408 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,408 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:34,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:34,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:34,408 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:34,408 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:34,409 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:34,409 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:34,409 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:34,409 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:34,409 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:34,409 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:34,409 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:34,409 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:34,409 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51e645e88856df580ad0de225d47b16a 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-20 00:56:34,409 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=A 2023-03-20 00:56:34,409 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:34,409 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=B 2023-03-20 00:56:34,410 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:34,410 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=C 2023-03-20 00:56:34,410 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:34,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:34,504 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:34,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:34,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273854513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,517 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:34,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273854514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:34,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273854516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:34,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273854516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,619 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:34,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273854617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,619 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:34,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273854617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,620 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:34,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273854619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,821 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:34,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273854820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:34,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273854821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,824 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:34,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273854822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,843 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1245 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/b924b94e8117445e80a133f74ffa7494 2023-03-20 00:56:34,851 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1245 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/9f8e5dd90e5e4761aabb69c38d25d4c3 2023-03-20 00:56:34,860 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1245 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/5d093e40c0984ab4b3efd9163c14fc89 2023-03-20 00:56:34,864 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/b924b94e8117445e80a133f74ffa7494 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/b924b94e8117445e80a133f74ffa7494 2023-03-20 00:56:34,869 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/b924b94e8117445e80a133f74ffa7494, entries=150, sequenceid=1245, filesize=11.9 K 2023-03-20 00:56:34,870 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/9f8e5dd90e5e4761aabb69c38d25d4c3 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/9f8e5dd90e5e4761aabb69c38d25d4c3 2023-03-20 00:56:34,874 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/9f8e5dd90e5e4761aabb69c38d25d4c3, entries=150, sequenceid=1245, filesize=11.9 K 2023-03-20 00:56:34,875 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/5d093e40c0984ab4b3efd9163c14fc89 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/5d093e40c0984ab4b3efd9163c14fc89 2023-03-20 00:56:34,879 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/5d093e40c0984ab4b3efd9163c14fc89, entries=150, sequenceid=1245, filesize=11.9 K 2023-03-20 00:56:34,880 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 51e645e88856df580ad0de225d47b16a in 471ms, sequenceid=1245, compaction requested=true 2023-03-20 00:56:34,880 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:34,880 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:34,880 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:34,880 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:34,880 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,880 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:34,880 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:34,881 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,881 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:34,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:34,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:34,882 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:34,882 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:34,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:34,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:34,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:34,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:34,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:34,883 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:34,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d1b8139[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:34,883 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:34,884 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:34,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:34,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:34,884 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:34,884 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:34,884 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:34,885 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,885 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:34,885 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,885 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:34,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:34,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:34,885 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:34,885 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:34,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:34,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:34,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:34,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:34,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:34,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:34,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:34,897 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:34,897 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:34,897 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:34,897 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:34,897 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:34,904 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:34,904 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:34,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:34,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:34,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:34,904 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:34,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:34,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:34,905 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:34,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:34,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:34,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:34,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:34,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:34,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:34,906 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:34,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:34,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:34,907 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51e645e88856df580ad0de225d47b16a 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-20 00:56:34,908 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=A 2023-03-20 00:56:34,908 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:34,908 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=B 2023-03-20 00:56:34,908 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:34,908 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=C 2023-03-20 00:56:34,908 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:34,908 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,908 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:34,908 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:34,908 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:34,908 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:34,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:34,908 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:34,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:34,908 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:34,908 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:34,908 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:34,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:34,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:34,908 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:34,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:34,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:34,909 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:34,909 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:34,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:34,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:34,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273854983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,009 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:35,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:35,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:35,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:35,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:35,012 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:35,012 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:35,012 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:35,012 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:35,012 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,012 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,013 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:35,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:35,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:35,013 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,013 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,013 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:35,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:35,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,014 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:35,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:35,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:35,014 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:35,014 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:35,014 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:35,014 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:35,014 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:35,014 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:35,014 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:35,014 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:35,014 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:35,015 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:35,015 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:35,015 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,015 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:35,015 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:35,016 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,016 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:35,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:35,016 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,016 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:35,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:35,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:35,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:35,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:35,017 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:35,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e38e4ab[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:35,017 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,018 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,018 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,018 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:35,018 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:35,018 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:35,018 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:35,019 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:35,019 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,019 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:35,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:35,019 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:35,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:35,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:35,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:35,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:35,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:35,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:35,021 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:35,021 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:35,021 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:35,021 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:35,022 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:35,023 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,023 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:35,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:35,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,023 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:35,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:35,023 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:35,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:35,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:35,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,024 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:35,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,026 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,026 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,026 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,026 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,026 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:35,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:35,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:35,026 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,026 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,026 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,027 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:35,027 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:35,027 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:35,027 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:35,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:35,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:35,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:35,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:35,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:35,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273855030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:35,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273855088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:35,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273855121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:35,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273855124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,127 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:35,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:35,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:35,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:35,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273855126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:35,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:35,129 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:35,129 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:35,129 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:35,129 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:35,129 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,129 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,130 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:35,130 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:35,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:35,130 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,130 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:35,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:35,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,131 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:35,131 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:35,131 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:35,131 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:35,131 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:35,132 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:35,132 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:35,132 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:35,132 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:35,132 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:35,132 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:35,132 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:35,132 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:35,132 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:35,132 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,132 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:35,132 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:35,133 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,133 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,133 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:35,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:35,133 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:35,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:35,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:35,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:35,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:35,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47179400[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:35,135 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:35,135 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,135 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,135 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:35,135 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:35,135 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,135 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:35,135 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:35,136 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,136 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:35,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:35,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:35,136 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:35,136 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:35,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:35,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:35,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:35,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:35,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:35,139 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:35,139 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:35,139 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:35,139 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:35,139 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:35,140 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:35,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:35,140 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,140 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:35,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:35,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:35,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:35,141 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,141 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:35,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,144 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,144 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,144 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:35,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:35,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:35,144 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:35,144 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,145 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,144 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:35,145 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,145 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:35,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:35,144 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:35,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:35,145 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:35,145 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:35,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:35,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:35,245 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:35,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:35,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:35,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:35,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:35,248 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:35,248 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:35,248 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:35,248 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:35,249 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,249 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,250 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:35,250 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:35,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:35,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,252 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,252 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:35,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:35,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:35,253 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:35,253 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:35,253 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:35,253 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:35,253 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:35,254 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:35,254 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:35,254 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:35,254 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:35,254 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:35,254 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:35,254 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:35,254 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:35,254 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,254 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:35,254 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:35,267 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,267 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:35,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:35,267 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,267 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:35,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:35,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:35,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:35,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:35,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@389b7b3f[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:35,271 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:35,271 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,280 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,280 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:35,280 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:35,280 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:35,280 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,280 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:35,283 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:35,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:35,284 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:35,284 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:35,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:35,284 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:35,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:35,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:35,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:35,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:35,287 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:35,287 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:35,287 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:35,287 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:35,287 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:35,292 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,292 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,292 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:35,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:35,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:35,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:35,292 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:35,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:35,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:35,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,294 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:35,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:35,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273855294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,305 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:35,305 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:35,305 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:35,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:35,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:35,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:35,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:35,305 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:35,305 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,305 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,305 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,305 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,305 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,305 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:35,305 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:35,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:35,305 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:35,316 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1259 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/382070e20f13462d8dc2280f17ba5500 2023-03-20 00:56:35,354 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1259 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/fb671629fddf4bf1be7712cc7a30f7a9 2023-03-20 00:56:35,405 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:35,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:35,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:35,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:35,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:35,411 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:35,411 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:35,411 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:35,411 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:35,415 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,415 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,415 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:35,415 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:35,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:35,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,416 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,416 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:35,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:35,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,417 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:35,417 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:35,417 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:35,417 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:35,417 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:35,417 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:35,417 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:35,417 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:35,417 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:35,417 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:35,417 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:35,417 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:35,417 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:35,417 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:35,417 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,417 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:35,418 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:35,418 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,418 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:35,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:35,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:35,419 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,419 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:35,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:35,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:35,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:35,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2998275c[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:35,423 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:35,423 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,425 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,425 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:35,425 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:35,425 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:35,425 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,425 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:35,426 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,426 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:35,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:35,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:35,426 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:35,426 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:35,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:35,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:35,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:35,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:35,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:35,428 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:35,428 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:35,428 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:35,428 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:35,428 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:35,429 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,429 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:35,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:35,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,429 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:35,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:35,430 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:35,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:35,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:35,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,431 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:35,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,432 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:35,432 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:35,432 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,432 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:35,432 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,432 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:35,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:35,432 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:35,433 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,433 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,433 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:35,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:35,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:35,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:35,434 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:35,434 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:35,534 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:35,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:35,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:35,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:35,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:35,537 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:35,537 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:35,537 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:35,537 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:35,538 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,538 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,538 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:35,538 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:35,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:35,539 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,539 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:35,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:35,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,539 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:35,539 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:35,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:35,540 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:35,540 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:35,540 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:35,540 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:35,540 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:35,540 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:35,540 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:35,540 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:35,541 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:35,541 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:35,541 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:35,541 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,541 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:35,541 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:35,542 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,542 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:35,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:35,542 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,542 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:35,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:35,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:35,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:35,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:35,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70dc8378[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:35,544 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:35,544 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,545 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,545 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:35,545 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,545 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:35,545 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:35,545 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:35,546 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:35,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:35,546 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:35,546 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:35,546 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:35,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:35,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:35,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:35,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:35,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:35,548 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:35,548 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:35,548 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:35,548 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:35,548 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:35,549 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,549 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:35,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,549 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:35,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:35,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:35,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:35,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:35,550 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:35,554 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:35,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,563 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:35,563 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:35,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:35,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:35,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:35,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:35,563 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:35,563 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,563 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,563 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,563 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:35,563 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:35,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:35,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:35,563 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:35,564 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,564 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,564 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:35,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273855601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:35,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273855628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:35,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273855629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:35,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273855630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,663 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:35,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:35,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:35,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:35,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:35,666 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:35,666 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:35,666 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:35,666 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:35,667 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,667 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,667 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,667 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:35,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:35,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:35,668 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,668 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:35,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:35,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:35,669 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:35,669 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:35,669 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:35,669 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:35,669 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:35,669 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:35,669 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:35,669 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:35,669 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:35,669 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:35,669 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:35,669 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:35,669 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:35,669 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,669 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:35,669 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:35,671 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,671 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:35,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:35,671 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,671 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:35,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:35,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:35,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:35,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:35,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b90aef7[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:35,672 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:35,672 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,673 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,673 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:35,673 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:35,673 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:35,673 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,673 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:35,674 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,674 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:35,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:35,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:35,674 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:35,674 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:35,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:35,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:35,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:35,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:35,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:35,675 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:35,675 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:35,675 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:35,675 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:35,675 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:35,676 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,676 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,676 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:35,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:35,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:35,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:35,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:35,677 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:35,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:35,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,678 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:35,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,679 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,679 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,679 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:35,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:35,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:35,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:35,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:35,679 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:35,680 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:35,680 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:35,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:35,679 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:35,679 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,681 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,681 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,681 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,681 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:35,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:35,779 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1259 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/7204e7b7e2f9425eb8a2e408fbade6ed 2023-03-20 00:56:35,780 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:35,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:35,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:35,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:35,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:35,783 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:35,783 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:35,783 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:35,783 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:35,783 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/382070e20f13462d8dc2280f17ba5500 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/382070e20f13462d8dc2280f17ba5500 2023-03-20 00:56:35,784 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,784 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,784 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:35,784 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:35,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:35,784 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,785 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:35,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:35,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,785 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:35,785 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:35,785 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:35,786 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:35,786 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:35,786 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:35,786 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:35,786 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:35,786 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:35,786 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:35,786 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:35,786 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:35,786 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:35,786 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:35,786 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,786 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:35,786 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:35,787 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/382070e20f13462d8dc2280f17ba5500, entries=200, sequenceid=1259, filesize=14.3 K 2023-03-20 00:56:35,787 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,787 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:35,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:35,787 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,787 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:35,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:35,788 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/fb671629fddf4bf1be7712cc7a30f7a9 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/fb671629fddf4bf1be7712cc7a30f7a9 2023-03-20 00:56:35,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:35,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:35,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:35,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68317e39[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:35,790 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:35,790 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,790 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,791 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:35,791 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,791 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:35,791 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:35,791 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:35,792 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/fb671629fddf4bf1be7712cc7a30f7a9, entries=150, sequenceid=1259, filesize=11.9 K 2023-03-20 00:56:35,792 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:35,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:35,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:35,792 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:35,792 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:35,792 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/7204e7b7e2f9425eb8a2e408fbade6ed as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/7204e7b7e2f9425eb8a2e408fbade6ed 2023-03-20 00:56:35,792 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:35,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:35,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:35,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:35,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:35,795 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:35,795 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:35,795 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:35,795 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:35,795 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:35,798 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/7204e7b7e2f9425eb8a2e408fbade6ed, entries=150, sequenceid=1259, filesize=11.9 K 2023-03-20 00:56:35,799 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 51e645e88856df580ad0de225d47b16a in 892ms, sequenceid=1259, compaction requested=true 2023-03-20 00:56:35,799 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:35,799 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:35,799 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:35,799 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-20 00:56:35,799 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-20 00:56:35,799 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:35,800 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 63610 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-20 00:56:35,800 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61170 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-20 00:56:35,800 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 51e645e88856df580ad0de225d47b16a/A is initiating minor compaction (all files) 2023-03-20 00:56:35,801 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 51e645e88856df580ad0de225d47b16a/B is initiating minor compaction (all files) 2023-03-20 00:56:35,801 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e645e88856df580ad0de225d47b16a/A in TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:35,801 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e645e88856df580ad0de225d47b16a/B in TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:35,801 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/c0939baa4fa64dd3b5e720389e45128c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/f8373e59a336466ea90c2c02c076b395, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/7e78a26f5bde488bac895b60289e8b12, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/b924b94e8117445e80a133f74ffa7494, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/382070e20f13462d8dc2280f17ba5500] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp, totalSize=62.1 K 2023-03-20 00:56:35,801 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/bd64cb659a56423ab60743556fc573da, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/fe22c3bd3d864e08b4a181bbb9ef43d3, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/079f4bb06b2a4e69a32dbd9b19253ab2, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/9f8e5dd90e5e4761aabb69c38d25d4c3, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/fb671629fddf4bf1be7712cc7a30f7a9] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp, totalSize=59.7 K 2023-03-20 00:56:35,801 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting c0939baa4fa64dd3b5e720389e45128c, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=1191, earliestPutTs=1679273790608 2023-03-20 00:56:35,801 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting bd64cb659a56423ab60743556fc573da, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=1191, earliestPutTs=1679273790608 2023-03-20 00:56:35,801 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting f8373e59a336466ea90c2c02c076b395, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1210, earliestPutTs=1679273791740 2023-03-20 00:56:35,802 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting fe22c3bd3d864e08b4a181bbb9ef43d3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1210, earliestPutTs=1679273791740 2023-03-20 00:56:35,802 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 7e78a26f5bde488bac895b60289e8b12, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1226, earliestPutTs=1679273792744 2023-03-20 00:56:35,802 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 079f4bb06b2a4e69a32dbd9b19253ab2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1226, earliestPutTs=1679273792744 2023-03-20 00:56:35,802 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting b924b94e8117445e80a133f74ffa7494, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1245, earliestPutTs=1679273793883 2023-03-20 00:56:35,802 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 382070e20f13462d8dc2280f17ba5500, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=1259, earliestPutTs=1679273794512 2023-03-20 00:56:35,803 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 9f8e5dd90e5e4761aabb69c38d25d4c3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1245, earliestPutTs=1679273793883 2023-03-20 00:56:35,805 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,805 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,805 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:35,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:35,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:35,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:35,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:35,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:35,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:35,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:35,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:35,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,809 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,810 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,810 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting fb671629fddf4bf1be7712cc7a30f7a9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1259, earliestPutTs=1679273794514 2023-03-20 00:56:35,817 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:35,817 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:35,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:35,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:35,817 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,817 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,817 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,817 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:35,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:35,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:35,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:35,817 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:35,817 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:35,817 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,817 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:35,818 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,817 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:35,822 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 51e645e88856df580ad0de225d47b16a#B#compaction#214 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:35,823 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51e645e88856df580ad0de225d47b16a#A#compaction#213 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:35,917 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:35,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:35,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:35,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:35,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:35,920 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:35,920 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:35,920 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:35,920 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:35,921 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,921 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,921 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:35,921 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:35,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:35,921 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:35,921 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:35,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:35,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:35,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:35,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:35,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:35,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:35,922 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:35,922 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:35,922 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:35,922 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:35,923 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:35,923 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:35,923 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:35,923 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51e645e88856df580ad0de225d47b16a 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-20 00:56:35,923 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=A 2023-03-20 00:56:35,923 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:35,923 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=B 2023-03-20 00:56:35,923 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:35,923 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=C 2023-03-20 00:56:35,923 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:35,931 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1277 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/49f179b5719b4df8b44eebf64d7209f5 2023-03-20 00:56:35,943 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1277 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/36b7864e1fc64c4490444c3dffbb87f6 2023-03-20 00:56:36,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:36,036 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:36,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:36,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273856063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,106 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:36,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273856105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,167 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:36,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273856166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,238 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/2cd5702c446d45d3b326dbb949e00ad5 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/2cd5702c446d45d3b326dbb949e00ad5 2023-03-20 00:56:36,241 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/dab575a7e3d44b4cb5f4a5ffb1541c6c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/dab575a7e3d44b4cb5f4a5ffb1541c6c 2023-03-20 00:56:36,244 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 51e645e88856df580ad0de225d47b16a/B of 51e645e88856df580ad0de225d47b16a into 2cd5702c446d45d3b326dbb949e00ad5(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:36,245 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:36,245 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a., storeName=51e645e88856df580ad0de225d47b16a/B, priority=11, startTime=1679273795799; duration=0sec 2023-03-20 00:56:36,245 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:36,245 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-20 00:56:36,246 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 51e645e88856df580ad0de225d47b16a/A of 51e645e88856df580ad0de225d47b16a into dab575a7e3d44b4cb5f4a5ffb1541c6c(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:36,246 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:36,246 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a., storeName=51e645e88856df580ad0de225d47b16a/A, priority=11, startTime=1679273795799; duration=0sec 2023-03-20 00:56:36,246 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:36,246 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61170 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-20 00:56:36,246 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 51e645e88856df580ad0de225d47b16a/C is initiating minor compaction (all files) 2023-03-20 00:56:36,246 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e645e88856df580ad0de225d47b16a/C in TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:36,246 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/d43d7fe6286a4996a12b0566a6534473, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/e1144cb1ff44420095aad397b51b89ae, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/8831a9714ff44778a8429cbd143177b2, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/5d093e40c0984ab4b3efd9163c14fc89, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/7204e7b7e2f9425eb8a2e408fbade6ed] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp, totalSize=59.7 K 2023-03-20 00:56:36,247 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting d43d7fe6286a4996a12b0566a6534473, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=1191, earliestPutTs=1679273790608 2023-03-20 00:56:36,247 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting e1144cb1ff44420095aad397b51b89ae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1210, earliestPutTs=1679273791740 2023-03-20 00:56:36,248 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 8831a9714ff44778a8429cbd143177b2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1226, earliestPutTs=1679273792744 2023-03-20 00:56:36,248 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 5d093e40c0984ab4b3efd9163c14fc89, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1245, earliestPutTs=1679273793883 2023-03-20 00:56:36,248 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 7204e7b7e2f9425eb8a2e408fbade6ed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1259, earliestPutTs=1679273794514 2023-03-20 00:56:36,288 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 51e645e88856df580ad0de225d47b16a#C#compaction#218 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:36,357 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1277 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/6417c6f279ad4fbd882903e3f8f66ffb 2023-03-20 00:56:36,362 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/49f179b5719b4df8b44eebf64d7209f5 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/49f179b5719b4df8b44eebf64d7209f5 2023-03-20 00:56:36,365 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/49f179b5719b4df8b44eebf64d7209f5, entries=150, sequenceid=1277, filesize=11.9 K 2023-03-20 00:56:36,366 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/36b7864e1fc64c4490444c3dffbb87f6 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/36b7864e1fc64c4490444c3dffbb87f6 2023-03-20 00:56:36,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:36,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273856368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,378 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/36b7864e1fc64c4490444c3dffbb87f6, entries=150, sequenceid=1277, filesize=11.9 K 2023-03-20 00:56:36,379 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/6417c6f279ad4fbd882903e3f8f66ffb as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/6417c6f279ad4fbd882903e3f8f66ffb 2023-03-20 00:56:36,385 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/6417c6f279ad4fbd882903e3f8f66ffb, entries=150, sequenceid=1277, filesize=11.9 K 2023-03-20 00:56:36,386 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 51e645e88856df580ad0de225d47b16a in 463ms, sequenceid=1277, compaction requested=false 2023-03-20 00:56:36,386 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:36,386 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:36,386 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:36,386 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:36,386 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,386 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:36,386 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:36,387 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:36,387 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:36,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:36,388 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:36,388 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:36,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:36,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:36,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:36,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:36,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b16a4[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:36,393 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:36,393 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:36,400 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:36,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:36,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:36,400 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:36,400 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:36,400 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:36,400 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,401 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:36,404 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,404 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:36,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:36,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:36,405 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:36,405 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:36,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:36,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:36,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:36,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:36,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:36,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:36,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:36,407 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:36,407 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:36,407 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:36,407 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:36,407 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:36,408 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:36,408 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:36,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:36,408 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:36,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:36,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:36,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:36,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:36,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:36,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:36,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:36,409 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:36,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:36,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:36,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:36,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,410 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:36,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:36,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,412 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,412 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:36,412 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:36,412 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:36,412 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:36,412 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,412 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:36,412 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:36,412 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:36,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:36,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:36,412 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:36,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:36,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:36,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:36,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:36,413 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:36,413 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:36,513 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:36,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:36,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:36,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:36,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:36,526 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:36,526 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:36,526 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:36,526 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:36,527 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:36,527 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:36,527 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,527 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:36,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:36,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:36,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:36,528 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,528 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:36,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:36,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:36,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:36,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:36,529 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:36,529 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:36,529 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:36,531 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:36,531 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:36,562 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:36,562 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:36,562 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51e645e88856df580ad0de225d47b16a 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-20 00:56:36,562 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=A 2023-03-20 00:56:36,562 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:36,563 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=B 2023-03-20 00:56:36,563 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:36,563 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=C 2023-03-20 00:56:36,563 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:36,588 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1293 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/075c8903933e4cf8808edb1c88d3ded3 2023-03-20 00:56:36,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:36,641 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:36,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:36,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273856686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:36,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273856688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:36,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273856688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:36,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273856688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,698 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/2e77589b52ee48729774bd3966e7dd73 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/2e77589b52ee48729774bd3966e7dd73 2023-03-20 00:56:36,702 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 51e645e88856df580ad0de225d47b16a/C of 51e645e88856df580ad0de225d47b16a into 2e77589b52ee48729774bd3966e7dd73(size=12.3 K), total size for store is 24.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:36,702 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:36,702 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a., storeName=51e645e88856df580ad0de225d47b16a/C, priority=11, startTime=1679273795799; duration=0sec 2023-03-20 00:56:36,702 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:36,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:36,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273856789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:36,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273856793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,796 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:36,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273856794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,796 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:36,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273856794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,993 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:36,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273856992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,996 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:36,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273856995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,999 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:36,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273856997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:36,999 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:36,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273856998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,032 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1293 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/f4f99edd764940d9982418583c3e6505 2023-03-20 00:56:37,115 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:37,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273857115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:37,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273857294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:37,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273857298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:37,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273857300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:37,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273857301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,446 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1293 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/6d8ff18b7f404a3a9f2c8091e9d85950 2023-03-20 00:56:37,450 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/075c8903933e4cf8808edb1c88d3ded3 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/075c8903933e4cf8808edb1c88d3ded3 2023-03-20 00:56:37,453 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/075c8903933e4cf8808edb1c88d3ded3, entries=150, sequenceid=1293, filesize=11.9 K 2023-03-20 00:56:37,454 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/f4f99edd764940d9982418583c3e6505 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/f4f99edd764940d9982418583c3e6505 2023-03-20 00:56:37,457 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/f4f99edd764940d9982418583c3e6505, entries=150, sequenceid=1293, filesize=11.9 K 2023-03-20 00:56:37,458 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/6d8ff18b7f404a3a9f2c8091e9d85950 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/6d8ff18b7f404a3a9f2c8091e9d85950 2023-03-20 00:56:37,461 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/6d8ff18b7f404a3a9f2c8091e9d85950, entries=150, sequenceid=1293, filesize=11.9 K 2023-03-20 00:56:37,461 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 51e645e88856df580ad0de225d47b16a in 899ms, sequenceid=1293, compaction requested=true 2023-03-20 00:56:37,462 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:37,462 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:37,462 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:37,462 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:37,462 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,462 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:37,462 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:37,463 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,463 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:37,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:37,463 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,463 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:37,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:37,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:37,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:37,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:37,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:37,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14288290[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:37,464 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:37,464 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,466 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,466 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:37,466 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:37,466 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:37,466 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,466 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:37,467 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,467 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:37,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:37,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:37,467 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:37,467 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:37,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:37,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:37,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:37,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:37,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:37,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:37,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:37,469 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:37,469 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:37,469 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:37,469 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:37,469 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:37,470 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:37,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:37,470 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,470 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:37,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:37,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:37,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:37,471 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:37,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:37,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:37,471 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:37,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:37,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,473 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:37,473 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:37,473 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:37,473 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:37,473 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:37,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:37,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:37,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:37,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:37,473 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:37,474 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:37,474 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,474 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,474 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,474 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,474 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:37,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:37,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:37,574 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:37,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:37,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:37,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:37,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:37,577 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:37,577 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:37,577 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:37,577 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:37,577 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,577 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:37,578 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:37,578 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:37,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:37,578 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,578 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:37,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:37,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:37,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:37,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:37,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:37,579 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:37,579 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:37,579 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:37,579 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:37,579 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:37,579 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:37,579 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:37,579 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51e645e88856df580ad0de225d47b16a 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-20 00:56:37,579 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=A 2023-03-20 00:56:37,579 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:37,579 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=B 2023-03-20 00:56:37,579 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:37,579 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=C 2023-03-20 00:56:37,580 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:37,586 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1312 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/1cf01d060eb541d2bc782ee67422c6c8 2023-03-20 00:56:37,598 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1312 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/e681ea22ba1a44e0975b4900b234b3cd 2023-03-20 00:56:37,635 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1312 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/9c5b7fa631314b9e8c9670c2ab1222b4 2023-03-20 00:56:37,640 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/1cf01d060eb541d2bc782ee67422c6c8 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/1cf01d060eb541d2bc782ee67422c6c8 2023-03-20 00:56:37,653 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/1cf01d060eb541d2bc782ee67422c6c8, entries=150, sequenceid=1312, filesize=11.9 K 2023-03-20 00:56:37,654 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/e681ea22ba1a44e0975b4900b234b3cd as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/e681ea22ba1a44e0975b4900b234b3cd 2023-03-20 00:56:37,657 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/e681ea22ba1a44e0975b4900b234b3cd, entries=150, sequenceid=1312, filesize=11.9 K 2023-03-20 00:56:37,658 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/9c5b7fa631314b9e8c9670c2ab1222b4 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/9c5b7fa631314b9e8c9670c2ab1222b4 2023-03-20 00:56:37,662 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/9c5b7fa631314b9e8c9670c2ab1222b4, entries=150, sequenceid=1312, filesize=11.9 K 2023-03-20 00:56:37,663 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for 51e645e88856df580ad0de225d47b16a in 84ms, sequenceid=1312, compaction requested=true 2023-03-20 00:56:37,663 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:37,663 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:37,663 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:37,663 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:37,663 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,663 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:37,663 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:37,664 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:37,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:37,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:37,664 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:37,665 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,665 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:37,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:37,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:37,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:37,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ac171f4[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:37,666 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:37,666 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,666 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,667 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:37,667 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:37,667 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:37,667 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,667 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:37,667 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:37,667 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:37,667 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,667 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:37,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:37,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:37,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:37,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:37,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:37,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:37,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:37,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:37,670 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:37,670 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:37,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:37,670 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:37,670 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:37,670 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:37,671 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,671 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,671 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:37,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:37,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:37,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:37,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:37,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:37,671 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:37,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:37,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:37,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,672 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:37,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:37,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,674 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:37,674 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:37,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:37,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:37,674 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:37,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:37,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:37,674 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,674 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:37,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:37,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:37,674 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:37,675 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,674 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:37,675 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:37,675 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,675 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,675 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:37,775 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:37,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:37,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:37,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:37,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:37,778 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:37,778 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:37,778 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:37,778 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:37,779 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,779 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:37,779 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,779 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:37,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:37,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:37,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:37,780 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,780 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:37,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:37,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:37,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:37,782 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:37,782 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:37,782 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:37,782 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:37,782 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:37,782 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:37,782 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:37,790 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:37,790 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:37,790 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:37,790 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:37,790 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:37,790 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:37,790 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,790 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:37,790 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:37,800 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,800 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:37,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:37,800 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:37,800 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:37,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:37,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:37,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:37,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:37,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79e20423[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:37,801 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:37,801 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,804 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,804 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,805 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:37,804 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:37,805 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:37,805 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:37,805 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:37,805 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,805 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:37,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:37,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:37,805 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:37,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:37,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:37,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:37,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:37,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:37,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:37,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:37,807 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:37,807 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:37,807 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:37,807 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:37,807 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:37,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:37,808 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,808 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:37,808 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51e645e88856df580ad0de225d47b16a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-20 00:56:37,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,808 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=A 2023-03-20 00:56:37,808 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:37,808 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=B 2023-03-20 00:56:37,809 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:37,809 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=C 2023-03-20 00:56:37,809 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:37,809 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:37,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:37,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:37,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:37,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:37,812 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:37,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:37,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:37,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,813 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:37,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:37,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,821 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,821 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:37,821 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:37,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:37,821 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:37,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:37,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:37,821 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:37,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:37,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:37,822 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:37,822 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:37,821 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:37,822 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,822 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:37,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:37,822 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,822 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679273797822, max timestamp=1679273797826Num uniques 859; 2023-03-20 00:56:37,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679273797822, max timestamp=1679273797826Num uniques 859; 2023-03-20 00:56:37,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 00:56:37,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 00:56:37,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679273797822, max timestamp=1679273797826Num uniques 859; 2023-03-20 00:56:37,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 00:56:37,847 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679273797839, max timestamp=1679273797845Num uniques 1018; 2023-03-20 00:56:37,847 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-20 00:56:37,847 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679273797839, max timestamp=1679273797845Num uniques 1018; 2023-03-20 00:56:37,847 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-20 00:56:37,847 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679273797839, max timestamp=1679273797845Num uniques 1018; 2023-03-20 00:56:37,847 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-20 00:56:37,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:37,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273857864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:37,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273857864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:37,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273857865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:37,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273857866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,922 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:37,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:37,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:37,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:37,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:37,924 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:37,925 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:37,925 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:37,925 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:37,925 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,925 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:37,926 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,926 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:37,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:37,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:37,926 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,926 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:37,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:37,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:37,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:37,927 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:37,927 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:37,927 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:37,927 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:37,927 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:37,927 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:37,927 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:37,927 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:37,927 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:37,927 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:37,927 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:37,927 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:37,927 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:37,927 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:37,927 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,927 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:37,927 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:37,928 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,928 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:37,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:37,928 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,928 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:37,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:37,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:37,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:37,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:37,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:37,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2174e38d[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:37,930 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:37,930 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,930 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,930 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:37,930 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:37,930 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,930 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:37,930 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:37,932 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:37,932 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,933 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:37,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:37,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:37,933 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:37,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:37,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:37,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:37,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:37,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:37,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:37,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:37,934 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:37,934 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:37,934 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:37,935 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:37,935 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:37,935 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,935 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:37,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:37,935 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:37,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,936 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:37,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:37,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:37,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:37,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:37,937 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:37,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:37,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:37,938 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,938 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:37,938 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:37,938 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:37,938 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:37,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:37,938 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:37,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:37,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:37,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:37,938 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,939 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,939 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:37,939 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:37,939 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:37,939 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:37,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:37,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:37,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:37,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:37,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273857969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,971 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:37,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273857969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:37,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273857970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:37,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:37,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273857970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,039 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:38,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:38,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:38,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:38,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:38,042 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:38,042 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:38,042 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:38,042 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:38,043 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,043 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,043 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:38,043 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:38,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:38,043 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,043 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:38,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:38,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,045 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:38,046 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:38,046 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:38,046 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:38,046 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:38,046 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:38,046 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:38,046 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:38,046 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:38,046 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:38,046 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:38,046 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:38,046 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:38,046 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:38,046 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,046 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:38,047 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:38,047 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,047 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:38,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:38,048 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,048 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:38,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:38,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:38,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:38,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:38,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7118c931[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:38,049 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:38,049 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,049 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,049 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:38,049 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:38,049 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:38,050 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,050 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:38,050 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:38,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:38,050 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:38,050 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:38,050 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:38,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:38,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:38,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:38,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:38,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:38,052 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:38,052 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:38,052 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:38,052 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:38,052 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:38,053 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:38,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:38,053 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,053 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:38,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:38,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:38,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:38,054 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:38,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,055 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:38,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,056 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,056 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,056 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,056 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:38,056 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,056 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:38,056 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:38,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:38,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:38,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:38,056 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:38,057 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,057 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,057 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:38,057 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:38,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:38,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:38,157 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:38,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:38,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:38,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:38,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:38,160 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:38,160 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:38,160 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:38,160 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:38,161 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,161 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,161 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,161 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:38,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:38,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:38,162 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,162 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:38,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:38,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,163 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:38,166 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:38,166 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:38,166 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:38,166 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:38,166 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:38,166 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:38,166 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:38,166 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:38,166 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:38,166 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:38,167 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:38,167 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:38,167 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:38,167 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,167 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:38,167 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:38,168 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,168 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:38,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:38,168 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,168 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:38,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:38,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:38,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:38,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:38,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60474837[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:38,170 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:38,170 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,170 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,171 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:38,171 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,171 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:38,171 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:38,171 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:38,172 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:38,172 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:38,172 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:38,172 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:38,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:38,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:38,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:38,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:38,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:38,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:38,175 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:38,175 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:38,175 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:38,175 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:38,175 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:38,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:38,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273858173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:38,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273858173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:38,176 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273858173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,176 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,176 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:38,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:38,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:38,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:38,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273858173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:38,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:38,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:38,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,177 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:38,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,179 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:38,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,181 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,181 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:38,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:38,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:38,181 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:38,181 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:38,181 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,182 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,182 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:38,182 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:38,182 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,182 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,182 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,182 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:38,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:38,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:38,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:38,216 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1323 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/aef15723df944c42928614a1168dfdb4 2023-03-20 00:56:38,282 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:38,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:38,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:38,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:38,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:38,294 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:38,294 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:38,294 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:38,294 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:38,295 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,295 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,295 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,295 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:38,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:38,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:38,296 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,296 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:38,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:38,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:38,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:38,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:38,305 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:38,305 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:38,305 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:38,306 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:38,306 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:38,306 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:38,306 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:38,306 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:38,306 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:38,306 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:38,306 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:38,306 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,306 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:38,306 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:38,316 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,316 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:38,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:38,317 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,317 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:38,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:38,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:38,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:38,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:38,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@96d8b00[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:38,318 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:38,318 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,322 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,322 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:38,322 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,322 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:38,322 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:38,322 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:38,323 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:38,323 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:38,323 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:38,323 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:38,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:38,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:38,326 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:38,326 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:38,326 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:38,326 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:38,326 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:38,327 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,327 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:38,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,327 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:38,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:38,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:38,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:38,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:38,328 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:38,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,329 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:38,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,331 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,331 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,331 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:38,331 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,331 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:38,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:38,331 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:38,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:38,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:38,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:38,331 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,331 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,331 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,331 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:38,331 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:38,332 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:38,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:38,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:38,432 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:38,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:38,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:38,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:38,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:38,434 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:38,435 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:38,435 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:38,435 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:38,435 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,435 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,435 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,435 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:38,436 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,436 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:38,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:38,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:38,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:38,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:38,437 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:38,437 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:38,437 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:38,437 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:38,437 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:38,437 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:38,437 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:38,437 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:38,437 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:38,437 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:38,438 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:38,438 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:38,438 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:38,438 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,438 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:38,438 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:38,438 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,438 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:38,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:38,439 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:38,439 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:38,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:38,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:38,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:38,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f03692b[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:38,440 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:38,440 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,441 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:38,441 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:38,441 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:38,441 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,441 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:38,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:38,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:38,441 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:38,442 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:38,442 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:38,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:38,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:38,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:38,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:38,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:38,444 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:38,444 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:38,444 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:38,444 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:38,444 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:38,445 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,445 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:38,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,445 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:38,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:38,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:38,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:38,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:38,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,446 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:38,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,448 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:38,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,458 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:38,458 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:38,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:38,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:38,458 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:38,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:38,458 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:38,458 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,459 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,459 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,459 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,459 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,459 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:38,459 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:38,459 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:38,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:38,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:38,478 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:38,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273858477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:38,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273858478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:38,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273858478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:38,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273858478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,559 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:38,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:38,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:38,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:38,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:38,570 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:38,570 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:38,570 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:38,570 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:38,571 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,571 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,572 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:38,572 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:38,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:38,572 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,572 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:38,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:38,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,573 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:38,573 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:38,573 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:38,573 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:38,573 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:38,573 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:38,573 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:38,573 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:38,573 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:38,573 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:38,573 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:38,573 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:38,573 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:38,573 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:38,573 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,573 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:38,573 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:38,574 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,574 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:38,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:38,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:38,575 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,575 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:38,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:38,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:38,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:38,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@386578bb[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:38,576 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:38,576 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,577 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,577 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:38,577 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:38,577 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:38,577 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,577 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:38,577 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:38,577 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,577 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:38,578 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:38,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:38,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:38,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:38,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:38,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:38,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:38,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:38,580 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:38,580 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:38,580 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:38,580 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:38,580 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:38,581 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,581 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,581 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:38,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:38,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:38,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:38,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:38,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:38,581 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:38,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,583 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:38,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,585 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,585 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,585 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:38,585 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:38,585 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:38,585 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:38,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:38,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:38,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:38,585 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,585 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,585 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,585 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,585 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:38,585 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:38,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:38,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:38,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:38,632 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1323 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/e82a009353bd4d508a714ce02e689d71 2023-03-20 00:56:38,644 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1323 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/11eac65952b34391937104ab314fd4f8 2023-03-20 00:56:38,673 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/aef15723df944c42928614a1168dfdb4 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/aef15723df944c42928614a1168dfdb4 2023-03-20 00:56:38,681 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/aef15723df944c42928614a1168dfdb4, entries=150, sequenceid=1323, filesize=11.9 K 2023-03-20 00:56:38,682 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/e82a009353bd4d508a714ce02e689d71 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/e82a009353bd4d508a714ce02e689d71 2023-03-20 00:56:38,685 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:38,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:38,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:38,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:38,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:38,693 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:38,693 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:38,693 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:38,693 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:38,693 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,693 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,694 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:38,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:38,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:38,694 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,695 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,695 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/e82a009353bd4d508a714ce02e689d71, entries=100, sequenceid=1323, filesize=9.5 K 2023-03-20 00:56:38,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,695 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:38,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:38,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,695 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:38,695 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/11eac65952b34391937104ab314fd4f8 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/11eac65952b34391937104ab314fd4f8 2023-03-20 00:56:38,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:38,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:38,696 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:38,696 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:38,696 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:38,696 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:38,696 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:38,696 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:38,696 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:38,696 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:38,696 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:38,697 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:38,697 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:38,697 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,697 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:38,697 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:38,698 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:38,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:38,698 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,699 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,699 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:38,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:38,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:38,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:38,700 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/11eac65952b34391937104ab314fd4f8, entries=100, sequenceid=1323, filesize=9.5 K 2023-03-20 00:56:38,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:38,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62b98970[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:38,700 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:38,700 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,701 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 51e645e88856df580ad0de225d47b16a in 893ms, sequenceid=1323, compaction requested=true 2023-03-20 00:56:38,701 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:38,701 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,701 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:38,701 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-20 00:56:38,701 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-20 00:56:38,701 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:38,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,701 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:38,701 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:38,702 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:38,702 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:38,701 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,702 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:38,703 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:38,703 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,703 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:38,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:38,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:38,703 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:38,703 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58896 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-20 00:56:38,703 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 51e645e88856df580ad0de225d47b16a/B is initiating minor compaction (all files) 2023-03-20 00:56:38,703 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e645e88856df580ad0de225d47b16a/B in TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:38,703 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/2cd5702c446d45d3b326dbb949e00ad5, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/36b7864e1fc64c4490444c3dffbb87f6, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/f4f99edd764940d9982418583c3e6505, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/e681ea22ba1a44e0975b4900b234b3cd, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/e82a009353bd4d508a714ce02e689d71] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp, totalSize=57.5 K 2023-03-20 00:56:38,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:38,704 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61336 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-20 00:56:38,704 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 51e645e88856df580ad0de225d47b16a/A is initiating minor compaction (all files) 2023-03-20 00:56:38,704 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e645e88856df580ad0de225d47b16a/A in TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:38,704 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/dab575a7e3d44b4cb5f4a5ffb1541c6c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/49f179b5719b4df8b44eebf64d7209f5, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/075c8903933e4cf8808edb1c88d3ded3, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/1cf01d060eb541d2bc782ee67422c6c8, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/aef15723df944c42928614a1168dfdb4] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp, totalSize=59.9 K 2023-03-20 00:56:38,704 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 2cd5702c446d45d3b326dbb949e00ad5, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=1259, earliestPutTs=1679273794514 2023-03-20 00:56:38,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:38,704 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting dab575a7e3d44b4cb5f4a5ffb1541c6c, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=1259, earliestPutTs=1679273794514 2023-03-20 00:56:38,705 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 36b7864e1fc64c4490444c3dffbb87f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1277, earliestPutTs=1679273794961 2023-03-20 00:56:38,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,705 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 49f179b5719b4df8b44eebf64d7209f5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1277, earliestPutTs=1679273794961 2023-03-20 00:56:38,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,705 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting f4f99edd764940d9982418583c3e6505, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1293, earliestPutTs=1679273796056 2023-03-20 00:56:38,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:38,705 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 075c8903933e4cf8808edb1c88d3ded3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1293, earliestPutTs=1679273796056 2023-03-20 00:56:38,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,706 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting e681ea22ba1a44e0975b4900b234b3cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1312, earliestPutTs=1679273796677 2023-03-20 00:56:38,706 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 1cf01d060eb541d2bc782ee67422c6c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1312, earliestPutTs=1679273796677 2023-03-20 00:56:38,706 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting e82a009353bd4d508a714ce02e689d71, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1323, earliestPutTs=1679273797806 2023-03-20 00:56:38,706 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting aef15723df944c42928614a1168dfdb4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1323, earliestPutTs=1679273797806 2023-03-20 00:56:38,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:38,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:38,708 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:38,708 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:38,708 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:38,708 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:38,708 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:38,717 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:38,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:38,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:38,717 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:38,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,718 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:38,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:38,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,720 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,720 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:38,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,722 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 51e645e88856df580ad0de225d47b16a#B#compaction#228 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:38,722 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:38,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:38,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:38,723 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:38,723 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:38,723 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,723 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,723 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,723 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,723 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,723 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:38,723 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,724 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51e645e88856df580ad0de225d47b16a#A#compaction#229 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:38,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:38,725 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:38,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:38,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:38,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:38,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:38,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:38,765 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/94024ddf3469441ea57eb02709286f43 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/94024ddf3469441ea57eb02709286f43 2023-03-20 00:56:38,769 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/8aa358ef68184298ac3c2d4ebd5a568a as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/8aa358ef68184298ac3c2d4ebd5a568a 2023-03-20 00:56:38,772 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 51e645e88856df580ad0de225d47b16a/B of 51e645e88856df580ad0de225d47b16a into 94024ddf3469441ea57eb02709286f43(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:38,773 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:38,773 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a., storeName=51e645e88856df580ad0de225d47b16a/B, priority=11, startTime=1679273798701; duration=0sec 2023-03-20 00:56:38,773 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:38,773 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-20 00:56:38,776 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58896 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-20 00:56:38,776 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 51e645e88856df580ad0de225d47b16a/C is initiating minor compaction (all files) 2023-03-20 00:56:38,776 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e645e88856df580ad0de225d47b16a/C in TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:38,776 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/2e77589b52ee48729774bd3966e7dd73, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/6417c6f279ad4fbd882903e3f8f66ffb, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/6d8ff18b7f404a3a9f2c8091e9d85950, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/9c5b7fa631314b9e8c9670c2ab1222b4, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/11eac65952b34391937104ab314fd4f8] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp, totalSize=57.5 K 2023-03-20 00:56:38,777 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 2e77589b52ee48729774bd3966e7dd73, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=1259, earliestPutTs=1679273794514 2023-03-20 00:56:38,777 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 6417c6f279ad4fbd882903e3f8f66ffb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1277, earliestPutTs=1679273794961 2023-03-20 00:56:38,777 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 6d8ff18b7f404a3a9f2c8091e9d85950, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1293, earliestPutTs=1679273796056 2023-03-20 00:56:38,778 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 9c5b7fa631314b9e8c9670c2ab1222b4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1312, earliestPutTs=1679273796677 2023-03-20 00:56:38,778 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 11eac65952b34391937104ab314fd4f8, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1323, earliestPutTs=1679273797806 2023-03-20 00:56:38,793 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 51e645e88856df580ad0de225d47b16a#C#compaction#230 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:38,795 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 51e645e88856df580ad0de225d47b16a/A of 51e645e88856df580ad0de225d47b16a into 8aa358ef68184298ac3c2d4ebd5a568a(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:38,795 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:38,795 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a., storeName=51e645e88856df580ad0de225d47b16a/A, priority=11, startTime=1679273798701; duration=0sec 2023-03-20 00:56:38,795 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:38,823 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:38,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:38,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:38,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:38,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:38,826 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:38,826 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:38,826 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:38,826 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:38,827 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,827 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,828 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:38,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:38,828 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:38,828 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,828 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:38,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:38,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:38,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:38,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:38,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:38,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:38,829 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:38,829 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:38,829 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:38,840 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:38,840 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:38,840 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:38,840 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:38,840 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51e645e88856df580ad0de225d47b16a 3/3 column families, dataSize=134.18 KB heapSize=379.45 KB 2023-03-20 00:56:38,840 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=A 2023-03-20 00:56:38,840 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:56:38,841 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=B 2023-03-20 00:56:38,841 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:56:38,841 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=C 2023-03-20 00:56:38,841 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:56:38,863 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1378 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/526e47eccbc341a0a50889458a9f0419 2023-03-20 00:56:38,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:38,985 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:38,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:38,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273858990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:38,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273858990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:38,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273858990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:38,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:38,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273858991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:39,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273859095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,096 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:39,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273859095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,096 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:39,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273859095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:39,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273859096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,131 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:39,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273859130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,133 DEBUG [Thread-1417] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a., hostname=jenkins-hbase20.apache.org,34553,1679273725199, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 00:56:39,210 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/5913a7848d454d1e8fddc74afef7b12e as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/5913a7848d454d1e8fddc74afef7b12e 2023-03-20 00:56:39,215 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 51e645e88856df580ad0de225d47b16a/C of 51e645e88856df580ad0de225d47b16a into 5913a7848d454d1e8fddc74afef7b12e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:39,216 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:39,216 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a., storeName=51e645e88856df580ad0de225d47b16a/C, priority=11, startTime=1679273798701; duration=0sec 2023-03-20 00:56:39,216 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:39,290 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1378 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/30b5eb7d165f484f87b2a71f9c1c0ed6 2023-03-20 00:56:39,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:39,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273859296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:39,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273859297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:39,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273859298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,302 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:39,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273859300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:39,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273859599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:39,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:39,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273859600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273859600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:39,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273859603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,704 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1378 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/261f2b7c2ca941d3a70fdf82ab5b34e3 2023-03-20 00:56:39,718 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/526e47eccbc341a0a50889458a9f0419 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/526e47eccbc341a0a50889458a9f0419 2023-03-20 00:56:39,723 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/526e47eccbc341a0a50889458a9f0419, entries=150, sequenceid=1378, filesize=11.9 K 2023-03-20 00:56:39,730 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/30b5eb7d165f484f87b2a71f9c1c0ed6 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/30b5eb7d165f484f87b2a71f9c1c0ed6 2023-03-20 00:56:39,734 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/30b5eb7d165f484f87b2a71f9c1c0ed6, entries=150, sequenceid=1378, filesize=11.9 K 2023-03-20 00:56:39,739 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/261f2b7c2ca941d3a70fdf82ab5b34e3 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/261f2b7c2ca941d3a70fdf82ab5b34e3 2023-03-20 00:56:39,745 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/261f2b7c2ca941d3a70fdf82ab5b34e3, entries=150, sequenceid=1378, filesize=11.9 K 2023-03-20 00:56:39,746 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~379.41 KB/388512, currentSize=46.96 KB/48090 for 51e645e88856df580ad0de225d47b16a in 906ms, sequenceid=1378, compaction requested=false 2023-03-20 00:56:39,746 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:39,746 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:39,746 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:39,746 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:39,746 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,747 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:39,747 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:39,747 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:39,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:39,748 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:39,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:39,748 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:39,748 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:39,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:39,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:39,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:39,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:39,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e92bac[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:39,750 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:39,750 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:39,751 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:39,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:39,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:39,751 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:39,751 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:39,751 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:39,751 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,751 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:39,752 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:39,752 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:39,752 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:39,752 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:39,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:39,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:39,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:39,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:39,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:39,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:39,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:39,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:39,756 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:39,756 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:39,756 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:39,756 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:39,756 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:39,758 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:39,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:39,758 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:39,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:39,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:39,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:39,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:39,758 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:39,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:39,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:39,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:39,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:39,759 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:39,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:39,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:39,760 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:39,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:39,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,762 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:39,762 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:39,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:39,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:39,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:39,762 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:39,762 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,762 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:39,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:39,762 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:39,762 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:39,763 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:39,763 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,763 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:39,763 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:39,763 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:39,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:39,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:39,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:39,863 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:39,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:39,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:39,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:39,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:39,867 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:39,867 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:39,867 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:39,867 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:39,867 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:39,867 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:39,868 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:39,868 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:39,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:39,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:39,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:39,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:39,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:39,869 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:39,869 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:39,869 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:39,869 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:39,869 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:39,869 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:39,870 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:39,870 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:39,870 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:39,870 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:39,870 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51e645e88856df580ad0de225d47b16a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-20 00:56:39,870 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=A 2023-03-20 00:56:39,870 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:39,870 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=B 2023-03-20 00:56:39,870 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:39,870 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=C 2023-03-20 00:56:39,870 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:39,874 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:39,874 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:40,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:40,105 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:40,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679273800111, max timestamp=1679273800121Num uniques 859; 2023-03-20 00:56:40,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679273800111, max timestamp=1679273800121Num uniques 859; 2023-03-20 00:56:40,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679273800111, max timestamp=1679273800121Num uniques 859; 2023-03-20 00:56:40,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-20 00:56:40,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-20 00:56:40,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-20 00:56:40,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:40,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 374 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273860139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:40,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:40,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273860140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:40,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:40,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273860140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:40,144 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:40,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 373 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273860141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:40,246 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:40,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273860244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:40,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:40,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 376 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273860244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:40,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:40,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273860244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:40,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:40,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 375 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273860244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:40,296 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1389 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/88ba241b942d42d7aa7fdb20121c8364 2023-03-20 00:56:40,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:40,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273860447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:40,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:40,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273860448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:40,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:40,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 377 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273860448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:40,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:40,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 378 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273860449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:40,712 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1389 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/86ade0c058414cf7b8971efc678b9f1c 2023-03-20 00:56:40,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:40,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 374 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273860749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:40,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:40,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 379 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273860749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:40,751 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:40,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273860750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:40,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:40,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 380 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273860751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,136 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1389 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/22c6f077ff1046e8bddfe7cd578f38c1 2023-03-20 00:56:41,151 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/88ba241b942d42d7aa7fdb20121c8364 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/88ba241b942d42d7aa7fdb20121c8364 2023-03-20 00:56:41,156 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/88ba241b942d42d7aa7fdb20121c8364, entries=100, sequenceid=1389, filesize=9.5 K 2023-03-20 00:56:41,157 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/86ade0c058414cf7b8971efc678b9f1c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/86ade0c058414cf7b8971efc678b9f1c 2023-03-20 00:56:41,161 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/86ade0c058414cf7b8971efc678b9f1c, entries=100, sequenceid=1389, filesize=9.5 K 2023-03-20 00:56:41,163 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/22c6f077ff1046e8bddfe7cd578f38c1 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/22c6f077ff1046e8bddfe7cd578f38c1 2023-03-20 00:56:41,168 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/22c6f077ff1046e8bddfe7cd578f38c1, entries=100, sequenceid=1389, filesize=9.5 K 2023-03-20 00:56:41,169 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 51e645e88856df580ad0de225d47b16a in 1299ms, sequenceid=1389, compaction requested=true 2023-03-20 00:56:41,169 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:41,169 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:41,169 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:41,169 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:41,169 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,169 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:41,169 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:41,170 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,170 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:41,171 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,171 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:41,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:41,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13b7404b[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:41,172 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:41,172 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,173 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,173 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:41,173 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:41,173 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:41,174 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,174 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:41,174 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:41,174 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:41,174 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:41,175 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:41,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:41,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:41,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:41,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:41,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:41,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:41,177 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:41,177 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:41,177 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:41,177 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:41,177 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:41,178 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,179 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:41,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:41,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:41,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,180 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:41,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:41,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:41,181 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:41,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,182 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:41,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,188 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:41,188 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:41,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:41,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:41,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:41,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:41,189 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,189 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:41,189 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:41,189 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,189 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,189 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,189 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,189 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:41,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:41,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:41,189 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:41,189 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:41,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:41,255 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51e645e88856df580ad0de225d47b16a 3/3 column families, dataSize=154.31 KB heapSize=430.69 KB 2023-03-20 00:56:41,255 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=A 2023-03-20 00:56:41,255 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:56:41,255 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=B 2023-03-20 00:56:41,255 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:56:41,255 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=C 2023-03-20 00:56:41,255 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:56:41,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:41,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273861259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:41,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 378 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273861259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:41,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 384 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273861262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:41,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273861266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,290 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:41,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:41,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:41,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:41,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:41,292 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:41,292 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:41,292 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:41,292 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:41,292 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,292 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:41,293 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:41,293 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:41,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:41,293 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,293 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:41,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:41,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:41,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:41,294 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:41,294 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:41,294 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:41,294 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:41,294 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:41,294 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:41,294 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:41,294 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:41,294 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:41,294 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:41,294 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:41,294 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:41,295 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:41,295 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:41,295 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,295 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:41,295 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:41,295 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,295 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:41,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:41,296 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:41,296 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:41,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:41,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:41,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:41,297 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:41,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45226[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:41,297 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,297 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,297 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:41,297 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:41,297 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,297 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:41,297 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:41,298 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,298 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:41,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:41,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:41,298 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:41,298 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:41,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:41,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:41,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:41,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:41,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:41,300 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:41,300 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:41,300 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:41,300 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:41,300 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:41,301 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,301 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,301 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:41,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:41,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:41,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:41,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:41,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:41,301 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:41,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,302 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:41,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,303 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,303 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:41,303 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:41,303 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:41,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:41,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:41,303 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:41,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:41,303 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:41,303 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:41,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:41,303 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,304 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:41,304 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,304 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,304 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:41,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:41,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:41,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:41,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:41,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273861364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 380 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273861364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:41,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 386 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273861365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:41,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 385 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273861369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,404 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:41,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:41,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:41,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:41,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:41,406 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:41,406 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:41,406 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:41,407 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:41,407 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,407 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:41,409 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,409 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:41,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:41,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:41,409 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,409 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:41,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:41,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:41,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:41,410 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:41,410 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:41,410 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:41,410 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:41,410 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:41,410 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:41,410 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:41,410 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:41,410 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:41,410 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:41,410 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:41,410 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:41,410 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:41,411 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:41,411 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,411 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:41,411 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:41,411 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,411 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:41,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:41,411 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,412 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:41,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:41,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:41,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:41,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:41,413 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:41,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ed8b95f[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:41,413 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,414 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,414 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:41,414 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:41,414 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:41,414 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,414 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:41,414 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:41,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:41,414 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:41,414 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:41,414 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:41,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:41,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:41,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:41,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:41,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:41,416 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:41,416 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:41,416 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:41,416 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:41,416 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:41,417 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,417 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:41,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:41,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,417 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:41,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:41,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:41,417 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:41,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:41,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,418 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:41,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,420 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:41,420 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:41,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:41,420 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:41,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:41,420 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:41,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:41,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:41,420 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:41,420 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:41,420 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:41,420 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:41,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:41,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:41,420 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,420 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,420 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,420 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,520 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:41,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:41,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:41,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:41,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:41,523 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:41,523 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:41,523 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:41,523 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:41,523 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,524 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:41,524 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,524 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:41,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:41,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:41,524 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,524 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:41,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:41,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:41,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:41,525 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:41,525 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:41,525 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:41,526 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:41,526 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:41,526 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:41,526 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:41,526 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:41,526 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:41,526 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:41,526 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:41,526 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:41,526 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:41,526 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:41,526 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,526 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:41,526 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:41,538 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:41,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:41,538 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:41,539 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,539 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:41,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:41,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:41,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:41,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60a25363[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:41,541 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:41,541 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,541 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,542 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:41,542 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:41,542 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:41,542 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,542 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:41,543 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,543 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:41,543 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:41,543 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:41,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:41,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:41,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:41,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:41,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:41,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:41,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:41,545 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:41,545 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:41,545 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:41,545 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:41,545 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:41,547 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,547 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,547 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:41,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:41,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:41,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:41,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:41,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:41,548 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:41,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,554 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:41,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,558 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,558 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:41,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:41,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:41,558 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:41,559 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:41,559 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:41,559 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:41,558 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:41,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:41,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:41,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:41,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:41,559 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:41,560 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,560 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,560 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,560 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:41,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:41,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 382 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273861566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,569 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:41,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273861567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,570 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:41,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 388 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273861567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:41,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273861572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,659 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:41,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:41,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:41,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:41,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:41,662 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:41,662 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:41,662 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:41,662 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:41,663 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,663 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:41,663 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:41,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:41,663 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:41,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:41,664 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,664 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:41,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:41,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:41,664 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:41,666 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:41,666 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:41,666 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:41,666 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:41,666 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:41,666 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:41,666 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:41,666 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:41,667 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:41,667 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:41,667 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:41,667 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:41,667 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:41,667 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,667 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:41,667 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:41,668 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:41,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:41,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:41,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:41,669 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,669 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,669 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:41,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:41,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:41,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@493ef224[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:41,671 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:41,671 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,671 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,672 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,672 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:41,672 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:41,672 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:41,672 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:41,673 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,673 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:41,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:41,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:41,673 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:41,673 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:41,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:41,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:41,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:41,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:41,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:41,677 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:41,677 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:41,677 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:41,677 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:41,677 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:41,688 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,688 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:41,688 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:41,688 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=1426 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/e5a7148e11e54c8da91e9a9161ae7ee5 2023-03-20 00:56:41,688 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:41,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:41,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:41,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:41,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:41,692 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:41,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,700 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,700 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:41,700 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:41,700 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:41,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:41,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:41,700 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:41,700 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:41,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:41,700 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,700 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:41,700 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:41,700 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,701 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:41,700 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:41,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:41,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:41,801 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:41,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:41,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:41,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:41,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:41,814 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:41,814 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:41,814 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:41,814 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:41,815 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,815 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:41,816 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:41,816 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:41,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:41,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:41,817 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,817 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:41,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:41,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:41,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:41,834 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:41,834 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:41,834 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:41,834 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:41,834 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:41,850 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:41,850 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:41,851 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:41,851 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:41,851 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:41,851 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:41,852 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:41,852 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:41,852 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,852 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:41,852 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:41,854 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:41,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:41,854 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:41,855 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,855 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:41,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:41,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b090c47[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:41,856 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:41,856 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,857 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,857 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:41,857 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:41,857 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:41,857 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,857 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:41,858 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:41,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:41,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:41,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,859 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:41,859 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:41,859 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:41,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:41,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:41,861 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:41,862 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:41,862 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:41,862 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:41,862 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:41,863 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,863 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:41,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:41,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,863 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:41,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:41,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:41,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:41,864 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:41,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,865 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:41,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,867 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:41,867 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,867 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:41,867 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:41,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:41,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:41,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:41,867 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:41,867 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:41,867 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,867 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,867 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,868 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:41,868 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:41,868 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:41,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:41,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:41,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:41,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 384 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273861868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:41,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273861870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:41,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 390 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273861872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:41,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273861876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,968 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:41,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:41,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:41,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:41,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:41,978 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:41,978 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:41,978 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:41,978 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:41,979 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,979 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:41,980 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,980 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:41,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:41,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:41,980 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,980 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:41,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:41,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:41,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:41,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:41,981 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:41,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:41,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:41,990 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:41,990 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:41,990 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:41,994 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:41,994 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:41,994 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:41,994 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:41,994 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:41,994 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:41,994 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:41,994 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:41,994 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,994 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:41,995 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:41,996 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,996 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:41,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:41,997 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,997 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:41,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:41,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:41,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:41,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:41,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:41,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@704238[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:41,998 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:41,998 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,999 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:41,999 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:41,999 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:41,999 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:41,999 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:41,999 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:42,000 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,000 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:42,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:42,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:42,000 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:42,000 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:42,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:42,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:42,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:42,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:42,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:42,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:42,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:42,002 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:42,002 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:42,002 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:42,002 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:42,002 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:42,003 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,003 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,003 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:42,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:42,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:42,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:42,003 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:42,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:42,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:42,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:42,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:42,007 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:42,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:42,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,010 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:42,010 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:42,010 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:42,010 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:42,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:42,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:42,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:42,010 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:42,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:42,010 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:42,010 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:42,010 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,010 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:42,010 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:42,010 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,010 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:42,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:42,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:42,099 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=1426 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/f8f8020b8ead44f48c6a1f7669ee3562 2023-03-20 00:56:42,111 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:42,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:42,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:42,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:42,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:42,126 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:42,126 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:42,126 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:42,126 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:42,127 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,127 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:42,128 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:42,128 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:42,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:42,128 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,128 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:42,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:42,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:42,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:42,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:42,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:42,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:42,154 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:42,162 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:42,162 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:42,162 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:42,162 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:42,163 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:42,163 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:42,163 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:42,164 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:42,164 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:42,164 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:42,164 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,164 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:42,164 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:42,166 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,166 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:42,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:42,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:42,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:42,167 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:42,167 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:42,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:42,167 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=1426 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/894588811dbd4abcbf60047cca5e6667 2023-03-20 00:56:42,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:42,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:42,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:42,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@723b201e[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:42,171 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:42,171 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:42,173 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/e5a7148e11e54c8da91e9a9161ae7ee5 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/e5a7148e11e54c8da91e9a9161ae7ee5 2023-03-20 00:56:42,175 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:42,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:42,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:42,175 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,175 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:42,175 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:42,176 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:42,176 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:42,177 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:42,177 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,177 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:42,177 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:42,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:42,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:42,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:42,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:42,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:42,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:42,180 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:42,180 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:42,180 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:42,180 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:42,180 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:42,182 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,182 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,182 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:42,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:42,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:42,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,183 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/e5a7148e11e54c8da91e9a9161ae7ee5, entries=150, sequenceid=1426, filesize=11.9 K 2023-03-20 00:56:42,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:42,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:42,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:42,184 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:42,184 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/f8f8020b8ead44f48c6a1f7669ee3562 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/f8f8020b8ead44f48c6a1f7669ee3562 2023-03-20 00:56:42,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:42,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:42,186 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:42,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:42,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,196 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,196 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:42,196 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:42,196 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:42,196 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:42,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:42,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:42,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:42,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:42,196 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:42,196 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,197 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:42,197 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:42,197 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:42,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:42,197 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:42,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:42,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:42,197 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,197 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/f8f8020b8ead44f48c6a1f7669ee3562, entries=150, sequenceid=1426, filesize=11.9 K 2023-03-20 00:56:42,198 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/894588811dbd4abcbf60047cca5e6667 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/894588811dbd4abcbf60047cca5e6667 2023-03-20 00:56:42,204 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/894588811dbd4abcbf60047cca5e6667, entries=150, sequenceid=1426, filesize=11.9 K 2023-03-20 00:56:42,205 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~430.64 KB/440976, currentSize=33.54 KB/34350 for 51e645e88856df580ad0de225d47b16a in 950ms, sequenceid=1426, compaction requested=true 2023-03-20 00:56:42,206 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:42,206 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:42,206 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 00:56:42,206 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 00:56:42,206 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:42,206 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:42,207 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46880 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 00:56:42,208 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 51e645e88856df580ad0de225d47b16a/B is initiating minor compaction (all files) 2023-03-20 00:56:42,208 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e645e88856df580ad0de225d47b16a/B in TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:42,208 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/94024ddf3469441ea57eb02709286f43, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/30b5eb7d165f484f87b2a71f9c1c0ed6, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/86ade0c058414cf7b8971efc678b9f1c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/f8f8020b8ead44f48c6a1f7669ee3562] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp, totalSize=45.8 K 2023-03-20 00:56:42,209 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46880 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 00:56:42,209 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 51e645e88856df580ad0de225d47b16a/A is initiating minor compaction (all files) 2023-03-20 00:56:42,209 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e645e88856df580ad0de225d47b16a/A in TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:42,209 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/8aa358ef68184298ac3c2d4ebd5a568a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/526e47eccbc341a0a50889458a9f0419, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/88ba241b942d42d7aa7fdb20121c8364, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/e5a7148e11e54c8da91e9a9161ae7ee5] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp, totalSize=45.8 K 2023-03-20 00:56:42,209 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 94024ddf3469441ea57eb02709286f43, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=1323, earliestPutTs=1679273796677 2023-03-20 00:56:42,210 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 8aa358ef68184298ac3c2d4ebd5a568a, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=1323, earliestPutTs=1679273796677 2023-03-20 00:56:42,210 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 526e47eccbc341a0a50889458a9f0419, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1378, earliestPutTs=1679273797863 2023-03-20 00:56:42,210 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 88ba241b942d42d7aa7fdb20121c8364, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1389, earliestPutTs=1679273798988 2023-03-20 00:56:42,211 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 30b5eb7d165f484f87b2a71f9c1c0ed6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1378, earliestPutTs=1679273797863 2023-03-20 00:56:42,211 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting e5a7148e11e54c8da91e9a9161ae7ee5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1426, earliestPutTs=1679273800139 2023-03-20 00:56:42,212 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 86ade0c058414cf7b8971efc678b9f1c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1389, earliestPutTs=1679273798988 2023-03-20 00:56:42,213 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting f8f8020b8ead44f48c6a1f7669ee3562, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1426, earliestPutTs=1679273800139 2023-03-20 00:56:42,228 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 51e645e88856df580ad0de225d47b16a#B#compaction#240 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:42,235 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51e645e88856df580ad0de225d47b16a#A#compaction#241 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:42,275 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/4d9f0c21a9a043029a9c02db2154dccf as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/4d9f0c21a9a043029a9c02db2154dccf 2023-03-20 00:56:42,281 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 51e645e88856df580ad0de225d47b16a/A of 51e645e88856df580ad0de225d47b16a into 4d9f0c21a9a043029a9c02db2154dccf(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:42,281 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:42,281 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a., storeName=51e645e88856df580ad0de225d47b16a/A, priority=12, startTime=1679273802206; duration=0sec 2023-03-20 00:56:42,281 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:42,282 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 00:56:42,283 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46880 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 00:56:42,283 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 51e645e88856df580ad0de225d47b16a/C is initiating minor compaction (all files) 2023-03-20 00:56:42,283 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e645e88856df580ad0de225d47b16a/C in TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:42,283 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/5913a7848d454d1e8fddc74afef7b12e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/261f2b7c2ca941d3a70fdf82ab5b34e3, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/22c6f077ff1046e8bddfe7cd578f38c1, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/894588811dbd4abcbf60047cca5e6667] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp, totalSize=45.8 K 2023-03-20 00:56:42,284 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 5913a7848d454d1e8fddc74afef7b12e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=1323, earliestPutTs=1679273796677 2023-03-20 00:56:42,284 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 261f2b7c2ca941d3a70fdf82ab5b34e3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1378, earliestPutTs=1679273797863 2023-03-20 00:56:42,285 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 22c6f077ff1046e8bddfe7cd578f38c1, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1389, earliestPutTs=1679273798988 2023-03-20 00:56:42,285 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 894588811dbd4abcbf60047cca5e6667, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1426, earliestPutTs=1679273800139 2023-03-20 00:56:42,295 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51e645e88856df580ad0de225d47b16a#C#compaction#242 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:42,297 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:42,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:42,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:42,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:42,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:42,314 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:42,314 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:42,314 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:42,314 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:42,314 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,314 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:42,315 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:42,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:42,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:42,315 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:42,316 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,316 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:42,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:42,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:42,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:42,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:42,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:42,317 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:42,317 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:42,317 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:42,322 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:42,322 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:42,322 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:42,322 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:42,323 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51e645e88856df580ad0de225d47b16a 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-20 00:56:42,323 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=A 2023-03-20 00:56:42,323 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:42,323 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=B 2023-03-20 00:56:42,323 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:42,323 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=C 2023-03-20 00:56:42,323 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:42,349 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=1435 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/aaa3277230034f33b9035b2fe37703f7 2023-03-20 00:56:42,376 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:42,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:42,390 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=1435 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/1222c7c1a4b245cc9f725724b6d263bc 2023-03-20 00:56:42,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679273802391, max timestamp=1679273802395Num uniques 859; 2023-03-20 00:56:42,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 00:56:42,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679273802391, max timestamp=1679273802395Num uniques 859; 2023-03-20 00:56:42,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 00:56:42,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679273802391, max timestamp=1679273802395Num uniques 859; 2023-03-20 00:56:42,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 00:56:42,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679273802411, max timestamp=1679273802416Num uniques 1068; 2023-03-20 00:56:42,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679273802411, max timestamp=1679273802416Num uniques 1068; 2023-03-20 00:56:42,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679273802411, max timestamp=1679273802416Num uniques 1068; 2023-03-20 00:56:42,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-03-20 00:56:42,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-03-20 00:56:42,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-03-20 00:56:42,476 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:42,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:42,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 403 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273862470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273862470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679273802460, max timestamp=1679273802475Num uniques 1077; 2023-03-20 00:56:42,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-03-20 00:56:42,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679273802460, max timestamp=1679273802475Num uniques 1077; 2023-03-20 00:56:42,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-03-20 00:56:42,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679273802460, max timestamp=1679273802475Num uniques 1077; 2023-03-20 00:56:42,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-03-20 00:56:42,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:42,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 410 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273862518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679273802506, max timestamp=1679273802520Num uniques 1086; 2023-03-20 00:56:42,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679273802506, max timestamp=1679273802520Num uniques 1086; 2023-03-20 00:56:42,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-20 00:56:42,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-20 00:56:42,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679273802506, max timestamp=1679273802520Num uniques 1086; 2023-03-20 00:56:42,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-20 00:56:42,582 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679273802560, max timestamp=1679273802580Num uniques 1045; 2023-03-20 00:56:42,582 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-20 00:56:42,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679273802560, max timestamp=1679273802580Num uniques 1045; 2023-03-20 00:56:42,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-20 00:56:42,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679273802560, max timestamp=1679273802580Num uniques 1045; 2023-03-20 00:56:42,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-20 00:56:42,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:42,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 411 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273862610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:42,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273862611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:42,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 436 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273862614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,626 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:42,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 412 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273862623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,655 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/97d28e361da441a3abc048fd6913d7a9 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/97d28e361da441a3abc048fd6913d7a9 2023-03-20 00:56:42,661 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 51e645e88856df580ad0de225d47b16a/B of 51e645e88856df580ad0de225d47b16a into 97d28e361da441a3abc048fd6913d7a9(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:42,661 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:42,662 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a., storeName=51e645e88856df580ad0de225d47b16a/B, priority=12, startTime=1679273802206; duration=0sec 2023-03-20 00:56:42,662 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:42,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:42,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 413 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273862716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:42,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273862716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:42,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 438 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273862720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,726 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/37ebecbeb4b94e27b2659047977469af as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/37ebecbeb4b94e27b2659047977469af 2023-03-20 00:56:42,733 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 51e645e88856df580ad0de225d47b16a/C of 51e645e88856df580ad0de225d47b16a into 37ebecbeb4b94e27b2659047977469af(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:42,733 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:42,734 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a., storeName=51e645e88856df580ad0de225d47b16a/C, priority=12, startTime=1679273802206; duration=0sec 2023-03-20 00:56:42,734 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:42,810 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=1435 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/5643fe50fd094b9f80cb921092b25a25 2023-03-20 00:56:42,816 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/aaa3277230034f33b9035b2fe37703f7 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/aaa3277230034f33b9035b2fe37703f7 2023-03-20 00:56:42,825 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/aaa3277230034f33b9035b2fe37703f7, entries=100, sequenceid=1435, filesize=9.5 K 2023-03-20 00:56:42,826 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/1222c7c1a4b245cc9f725724b6d263bc as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/1222c7c1a4b245cc9f725724b6d263bc 2023-03-20 00:56:42,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:42,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 414 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273862827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,835 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/1222c7c1a4b245cc9f725724b6d263bc, entries=100, sequenceid=1435, filesize=9.5 K 2023-03-20 00:56:42,837 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/5643fe50fd094b9f80cb921092b25a25 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/5643fe50fd094b9f80cb921092b25a25 2023-03-20 00:56:42,845 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/5643fe50fd094b9f80cb921092b25a25, entries=100, sequenceid=1435, filesize=9.5 K 2023-03-20 00:56:42,845 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 51e645e88856df580ad0de225d47b16a in 522ms, sequenceid=1435, compaction requested=false 2023-03-20 00:56:42,846 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:42,846 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:42,846 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:42,846 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:42,846 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,846 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:42,846 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:42,847 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:42,847 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:42,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:42,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:42,847 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:42,848 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:42,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:42,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:42,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:42,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:42,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c2a43af[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:42,849 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:42,849 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:42,850 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:42,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:42,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:42,850 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:42,851 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:42,851 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:42,851 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,851 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:42,853 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:42,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:42,853 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:42,853 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:42,853 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:42,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:42,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:42,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:42,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:42,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:42,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:42,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:42,856 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:42,856 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:42,856 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:42,856 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:42,856 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:42,857 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:42,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:42,857 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,857 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:42,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:42,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:42,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:42,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:42,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:42,859 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:42,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:42,861 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,863 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,863 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:42,863 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:42,863 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,863 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:42,863 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:42,863 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,863 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:42,863 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:42,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:42,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:42,864 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:42,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:42,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:42,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:42,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:42,864 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:42,864 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:42,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:42,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679273802608, max timestamp=1679273802919Num uniques 1104; 2023-03-20 00:56:42,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-20 00:56:42,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679273802608, max timestamp=1679273802919Num uniques 1104; 2023-03-20 00:56:42,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-20 00:56:42,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679273802608, max timestamp=1679273802919Num uniques 1104; 2023-03-20 00:56:42,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-20 00:56:42,921 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51e645e88856df580ad0de225d47b16a 3/3 column families, dataSize=26.84 KB heapSize=66.70 KB 2023-03-20 00:56:42,922 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=A 2023-03-20 00:56:42,922 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:56:42,922 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=B 2023-03-20 00:56:42,922 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:56:42,922 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=C 2023-03-20 00:56:42,922 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:56:42,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679273802937, max timestamp=1679273802948Num uniques 859; 2023-03-20 00:56:42,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 00:56:42,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679273802937, max timestamp=1679273802948Num uniques 859; 2023-03-20 00:56:42,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 00:56:42,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679273802937, max timestamp=1679273802948Num uniques 859; 2023-03-20 00:56:42,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 00:56:42,964 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:42,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:42,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:42,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:42,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:42,968 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:42,969 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:42,969 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:42,969 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:42,969 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,969 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:42,970 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:42,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:42,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:42,970 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:42,970 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,970 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:42,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:42,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:42,972 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:42,978 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:42,978 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:42,978 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:42,978 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:42,978 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:42,979 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:42,979 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:42,979 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:42,979 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:42,979 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:42,979 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:42,979 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:42,979 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:42,979 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,979 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:42,979 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:42,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679273802963, max timestamp=1679273802982Num uniques 1068; 2023-03-20 00:56:42,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-03-20 00:56:42,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679273802963, max timestamp=1679273802982Num uniques 1068; 2023-03-20 00:56:42,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-03-20 00:56:42,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679273802963, max timestamp=1679273802982Num uniques 1068; 2023-03-20 00:56:42,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-03-20 00:56:42,985 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:42,985 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:42,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:42,985 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:42,985 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:42,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:42,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:42,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:42,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:42,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:42,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fe21559[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:42,987 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:42,987 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:42,988 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:42,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:42,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:42,988 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:42,988 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:42,989 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:42,989 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,989 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:42,989 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:42,990 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:42,990 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:42,990 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:42,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:42,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:42,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:42,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:42,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:42,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:42,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:42,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:42,993 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:42,993 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:42,993 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:42,993 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:42,993 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:42,994 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,994 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:42,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:42,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:42,995 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:42,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:42,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:42,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:42,996 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:42,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:42,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:42,998 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:42,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:43,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,002 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:43,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:43,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:43,002 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:43,002 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:43,002 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:43,002 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:43,002 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,002 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:43,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:43,002 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:43,003 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,003 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,003 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,003 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:43,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:43,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679273802996, max timestamp=1679273803016Num uniques 1077; 2023-03-20 00:56:43,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-03-20 00:56:43,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679273802996, max timestamp=1679273803016Num uniques 1077; 2023-03-20 00:56:43,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-03-20 00:56:43,023 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679273802996, max timestamp=1679273803016Num uniques 1077; 2023-03-20 00:56:43,023 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=6870, suffix heap size=139976,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=150 2023-03-20 00:56:43,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8506, totalHeapSize=19584, min timestamp=1679273803039, max timestamp=1679273803049Num uniques 1086; 2023-03-20 00:56:43,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679273803039, max timestamp=1679273803049Num uniques 1036; 2023-03-20 00:56:43,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49726, new segment data size=8506, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-03-20 00:56:43,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-20 00:56:43,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679273803039, max timestamp=1679273803049Num uniques 1009; 2023-03-20 00:56:43,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8516, suffix heap size=132904,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=186 2023-03-20 00:56:43,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679273803068, max timestamp=1679273803078Num uniques 1045; 2023-03-20 00:56:43,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-20 00:56:43,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679273803068, max timestamp=1679273803078Num uniques 1045; 2023-03-20 00:56:43,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-20 00:56:43,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679273803068, max timestamp=1679273803078Num uniques 1045; 2023-03-20 00:56:43,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-20 00:56:43,102 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:43,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679273803093, max timestamp=1679273803101Num uniques 1104; 2023-03-20 00:56:43,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-20 00:56:43,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:43,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679273803093, max timestamp=1679273803101Num uniques 1104; 2023-03-20 00:56:43,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679273803093, max timestamp=1679273803101Num uniques 1104; 2023-03-20 00:56:43,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-20 00:56:43,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-20 00:56:43,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:43,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:43,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:43,105 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:43,106 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:43,106 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:43,106 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:43,106 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,106 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,107 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,107 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:43,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:43,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:43,107 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,107 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:43,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:43,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,108 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:43,108 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:43,108 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:43,109 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:43,109 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:43,109 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:43,109 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:43,109 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:43,109 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:43,109 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:43,109 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:43,109 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:43,109 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:43,109 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:43,109 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,109 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:43,109 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:43,110 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,110 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:43,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:43,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:43,111 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,111 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:43,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:43,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:43,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:43,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d81eb1f[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:43,114 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:43,114 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,115 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,115 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:43,115 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:43,115 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:43,115 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,115 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:43,117 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:43,117 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:43,117 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:43,117 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:43,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:43,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:43,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:43,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:43,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:43,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:43,121 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:43,121 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:43,121 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:43,121 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:43,121 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:43,125 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,125 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,125 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:43,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:43,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:43,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:43,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:43,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:43,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,126 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:43,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,129 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:43,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,133 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,133 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,133 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,133 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,133 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,133 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,134 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,134 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:43,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:43,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:43,133 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:43,134 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:43,134 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:43,134 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:43,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:43,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:43,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:43,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:43,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1679273803121, max timestamp=1679273803132Num uniques 1013; 2023-03-20 00:56:43,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=12038, suffix heap size=133320,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=263 2023-03-20 00:56:43,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1679273803121, max timestamp=1679273803132Num uniques 1013; 2023-03-20 00:56:43,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=12038, suffix heap size=133320,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=263 2023-03-20 00:56:43,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679273803129, max timestamp=1679273803132Num uniques 1013; 2023-03-20 00:56:43,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=6870, suffix heap size=133320,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=150 2023-03-20 00:56:43,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679273803140, max timestamp=1679273803167Num uniques 1122; 2023-03-20 00:56:43,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679273803140, max timestamp=1679273803167Num uniques 1122; 2023-03-20 00:56:43,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=10162, suffix heap size=144656,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=222 2023-03-20 00:56:43,171 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679273803140, max timestamp=1679273803167Num uniques 1009; 2023-03-20 00:56:43,171 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10162, suffix heap size=132904,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=222 2023-03-20 00:56:43,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=10162, suffix heap size=144656,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=222 2023-03-20 00:56:43,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1679273803183, max timestamp=1679273803190Num uniques 1081; 2023-03-20 00:56:43,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1679273803183, max timestamp=1679273803190Num uniques 1081; 2023-03-20 00:56:43,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-03-20 00:56:43,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-03-20 00:56:43,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679273803187, max timestamp=1679273803190Num uniques 1081; 2023-03-20 00:56:43,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-03-20 00:56:43,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1679273803206, max timestamp=1679273803216Num uniques 1140; 2023-03-20 00:56:43,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=15570, suffix heap size=146528,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=340 2023-03-20 00:56:43,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679273803206, max timestamp=1679273803216Num uniques 1140; 2023-03-20 00:56:43,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=8700, suffix heap size=146528,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=190 2023-03-20 00:56:43,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679273803206, max timestamp=1679273803216Num uniques 1040; 2023-03-20 00:56:43,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-03-20 00:56:43,238 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:43,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:43,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:43,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:43,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:43,242 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:43,242 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:43,242 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:43,242 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:43,243 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,243 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=499, cellSize=22854, totalHeapSize=52136, min timestamp=1679273803227, max timestamp=1679273803241Num uniques 1049; 2023-03-20 00:56:43,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=22854, suffix heap size=137064,new segment heap size=52136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=499 2023-03-20 00:56:43,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=499, cellSize=22854, totalHeapSize=52136, min timestamp=1679273803227, max timestamp=1679273803241Num uniques 1049; 2023-03-20 00:56:43,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=499, cellSize=22854, totalHeapSize=52136, min timestamp=1679273803227, max timestamp=1679273803241Num uniques 1199; 2023-03-20 00:56:43,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=22854, suffix heap size=137064,new segment heap size=52136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=499 2023-03-20 00:56:43,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=22854, suffix heap size=152664,new segment heap size=52136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=499 2023-03-20 00:56:43,245 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,245 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:43,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:43,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:43,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,246 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:43,250 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:43,250 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:43,250 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:43,250 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:43,250 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:43,250 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,250 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:43,254 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:43,254 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:43,254 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:43,254 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:43,254 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:43,254 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:43,254 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:43,254 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:43,255 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,255 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:43,255 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:43,258 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,258 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:43,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:43,259 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,259 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:43,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:43,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:43,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:43,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:43,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46911d61[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:43,262 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:43,262 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,262 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,262 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:43,263 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:43,263 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:43,263 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,263 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:43,264 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:43,264 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:43,264 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:43,264 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:43,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:43,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:43,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:43,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:43,267 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:43,267 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:43,267 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:43,268 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:43,268 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:43,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:43,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 475 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273863262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:43,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:43,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273863262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 496 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273863264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:43,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 385 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273863264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:43,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 434 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273863267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,272 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,272 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:43,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:43,272 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:43,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:43,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:43,273 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:43,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:43,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,275 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:43,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,284 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,284 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:43,284 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:43,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:43,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:43,284 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:43,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:43,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:43,284 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:43,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:43,285 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:43,285 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:43,285 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,285 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,285 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,285 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,285 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:43,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 477 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273863370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:43,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 498 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273863370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:43,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273863370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:43,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:43,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273863370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 436 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273863370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,385 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:43,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:43,387 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1544 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/5b2c6ba1546f4dcc92447a98b8f4fecb 2023-03-20 00:56:43,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:43,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:43,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:43,394 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:43,394 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:43,394 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:43,394 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:43,394 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,395 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,395 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:43,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:43,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:43,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,396 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,396 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,396 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:43,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:43,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:43,398 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:43,398 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:43,398 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:43,398 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:43,398 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:43,413 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:43,413 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:43,413 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:43,413 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:43,413 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:43,413 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:43,413 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:43,413 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:43,413 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,430 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:43,430 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:43,431 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,431 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:43,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:43,432 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,432 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:43,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:43,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:43,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:43,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:43,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58b80927[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:43,433 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:43,433 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,434 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:43,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:43,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:43,434 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,434 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:43,435 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:43,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:43,435 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:43,435 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:43,435 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:43,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:43,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:43,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:43,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:43,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:43,438 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:43,438 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:43,438 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:43,438 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:43,438 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:43,440 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,440 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:43,440 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,440 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:43,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:43,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:43,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:43,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:43,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:43,442 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:43,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,444 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:43,444 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:43,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:43,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:43,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:43,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:43,444 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,445 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,445 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:43,445 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,445 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,445 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,445 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,445 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:43,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:43,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:43,445 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:43,445 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:43,456 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1544 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/e6357ddb720541478a71a67878fda858 2023-03-20 00:56:43,545 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:43,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:43,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:43,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:43,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:43,552 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:43,552 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:43,552 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:43,552 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:43,552 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,552 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,553 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:43,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:43,553 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:43,554 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,554 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:43,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:43,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:43,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:43,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:43,562 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:43,562 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:43,562 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:43,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:43,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 479 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273863573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:43,575 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:43,576 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:43,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273863573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,576 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:43,576 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:43,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:43,576 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:43,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273863575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,576 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:43,576 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:43,576 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:43,576 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,576 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:43,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:43,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 500 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273863575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,576 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:43,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:43,577 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 438 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273863575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:43,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:43,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:43,577 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:43,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,579 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,579 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:43,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:43,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:43,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@270f87b4[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:43,580 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:43,580 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,581 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,581 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,581 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:43,581 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:43,581 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:43,581 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:43,584 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:43,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:43,585 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:43,585 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:43,585 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:43,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:43,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:43,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:43,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:43,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:43,587 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:43,587 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:43,587 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:43,587 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:43,587 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:43,597 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,597 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,597 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:43,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:43,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:43,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:43,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:43,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:43,597 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:43,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,599 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:43,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,600 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:43,600 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:43,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:43,601 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:43,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:43,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:43,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:43,601 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:43,601 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:43,601 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,601 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,601 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,601 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,601 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,601 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,601 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:43,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:43,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:43,701 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:43,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:43,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:43,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:43,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:43,704 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:43,704 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:43,704 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:43,704 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:43,704 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,704 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,705 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:43,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:43,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:43,705 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,705 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,705 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:43,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:43,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,706 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:43,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:43,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:43,706 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:43,706 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:43,706 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:43,706 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:43,706 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:43,706 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:43,707 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:43,707 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:43,707 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:43,707 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:43,707 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:43,707 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,707 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:43,707 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:43,707 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,707 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:43,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:43,708 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,708 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:43,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:43,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:43,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:43,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:43,709 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:43,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45344959[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:43,709 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,709 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,710 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:43,710 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:43,710 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:43,710 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,710 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:43,710 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,710 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:43,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:43,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:43,710 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:43,711 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:43,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:43,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:43,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:43,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:43,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:43,713 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:43,713 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:43,713 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:43,713 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:43,713 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:43,714 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,714 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:43,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:43,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,714 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:43,714 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:43,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:43,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:43,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:43,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,715 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:43,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,717 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,717 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:43,717 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:43,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:43,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:43,717 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,717 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,717 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,717 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,717 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,717 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:43,717 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:43,717 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:43,717 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:43,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:43,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:43,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:43,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:43,818 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:43,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:43,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:43,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:43,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:43,820 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:43,820 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:43,820 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:43,820 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:43,821 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,821 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,821 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,821 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:43,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:43,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:43,822 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,822 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:43,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:43,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,822 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:43,822 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:43,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:43,823 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:43,823 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:43,823 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:43,823 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:43,823 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:43,823 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:43,823 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:43,823 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:43,823 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:43,823 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:43,823 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:43,823 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,823 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:43,823 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:43,824 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,824 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:43,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:43,824 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,824 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:43,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:43,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:43,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:43,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:43,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b003e8b[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:43,825 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:43,825 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,826 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,826 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:43,826 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:43,826 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:43,826 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,826 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:43,827 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,827 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:43,827 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:43,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:43,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:43,827 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:43,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:43,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:43,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:43,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:43,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:43,829 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:43,829 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:43,829 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:43,829 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:43,829 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:43,830 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,830 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,830 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:43,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:43,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:43,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:43,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:43,830 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:43,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:43,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,831 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:43,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:43,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,833 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:43,833 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:43,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:43,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:43,833 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:43,833 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:43,833 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:43,833 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,833 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:43,833 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,833 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,833 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:43,833 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:43,833 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,880 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:43,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 481 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273863877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,880 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:43,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273863877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,880 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:43,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 502 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273863878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,881 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:43,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273863879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,881 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:43,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 440 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273863879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,897 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1544 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/44af2bb3ea4549eda3b81a9bebd91e7e 2023-03-20 00:56:43,900 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/5b2c6ba1546f4dcc92447a98b8f4fecb as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/5b2c6ba1546f4dcc92447a98b8f4fecb 2023-03-20 00:56:43,904 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/5b2c6ba1546f4dcc92447a98b8f4fecb, entries=150, sequenceid=1544, filesize=11.9 K 2023-03-20 00:56:43,904 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/e6357ddb720541478a71a67878fda858 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/e6357ddb720541478a71a67878fda858 2023-03-20 00:56:43,907 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/e6357ddb720541478a71a67878fda858, entries=150, sequenceid=1544, filesize=11.9 K 2023-03-20 00:56:43,908 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/44af2bb3ea4549eda3b81a9bebd91e7e as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/44af2bb3ea4549eda3b81a9bebd91e7e 2023-03-20 00:56:43,911 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/44af2bb3ea4549eda3b81a9bebd91e7e, entries=150, sequenceid=1544, filesize=11.9 K 2023-03-20 00:56:43,912 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~66.66 KB/68256, currentSize=167.72 KB/171750 for 51e645e88856df580ad0de225d47b16a in 991ms, sequenceid=1544, compaction requested=true 2023-03-20 00:56:43,912 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:43,912 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:43,912 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:56:43,912 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:56:43,912 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:43,912 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:43,913 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34830 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:56:43,913 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 51e645e88856df580ad0de225d47b16a/B is initiating minor compaction (all files) 2023-03-20 00:56:43,913 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34830 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:56:43,913 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e645e88856df580ad0de225d47b16a/B in TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:43,913 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 51e645e88856df580ad0de225d47b16a/A is initiating minor compaction (all files) 2023-03-20 00:56:43,913 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/97d28e361da441a3abc048fd6913d7a9, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/1222c7c1a4b245cc9f725724b6d263bc, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/e6357ddb720541478a71a67878fda858] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp, totalSize=34.0 K 2023-03-20 00:56:43,913 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e645e88856df580ad0de225d47b16a/A in TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:43,913 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/4d9f0c21a9a043029a9c02db2154dccf, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/aaa3277230034f33b9035b2fe37703f7, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/5b2c6ba1546f4dcc92447a98b8f4fecb] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp, totalSize=34.0 K 2023-03-20 00:56:43,913 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 97d28e361da441a3abc048fd6913d7a9, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=1426, earliestPutTs=1679273800139 2023-03-20 00:56:43,913 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 4d9f0c21a9a043029a9c02db2154dccf, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=1426, earliestPutTs=1679273800139 2023-03-20 00:56:43,913 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 1222c7c1a4b245cc9f725724b6d263bc, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1435, earliestPutTs=1679273801259 2023-03-20 00:56:43,913 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting aaa3277230034f33b9035b2fe37703f7, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1435, earliestPutTs=1679273801259 2023-03-20 00:56:43,914 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting e6357ddb720541478a71a67878fda858, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1544, earliestPutTs=1679273802608 2023-03-20 00:56:43,914 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 5b2c6ba1546f4dcc92447a98b8f4fecb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1544, earliestPutTs=1679273802608 2023-03-20 00:56:43,918 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 51e645e88856df580ad0de225d47b16a#B#compaction#249 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:43,919 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51e645e88856df580ad0de225d47b16a#A#compaction#250 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:43,933 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:43,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:43,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:43,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:43,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:43,938 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:43,938 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:43,938 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:43,938 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:43,938 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,938 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,939 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:43,939 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:43,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:43,939 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:43,939 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:43,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:43,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:43,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:43,940 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:43,941 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:43,941 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:43,941 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:43,941 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:43,941 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:43,942 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:43,942 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:43,942 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:43,942 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:43,942 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51e645e88856df580ad0de225d47b16a 3/3 column families, dataSize=167.72 KB heapSize=452.72 KB 2023-03-20 00:56:43,942 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=A 2023-03-20 00:56:43,942 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:56:43,942 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=B 2023-03-20 00:56:43,943 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:56:43,943 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=C 2023-03-20 00:56:43,943 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:56:43,952 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=1751 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/9d7514ef388145f6b5e4090644049052 2023-03-20 00:56:44,340 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/e30fd4dcecc84f39bd3e717dc3b4f8fe as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/e30fd4dcecc84f39bd3e717dc3b4f8fe 2023-03-20 00:56:44,342 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/876ce9e9f2c243d7bdac9856ee4ea97c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/876ce9e9f2c243d7bdac9856ee4ea97c 2023-03-20 00:56:44,348 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 51e645e88856df580ad0de225d47b16a/B of 51e645e88856df580ad0de225d47b16a into e30fd4dcecc84f39bd3e717dc3b4f8fe(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:44,348 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:44,348 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a., storeName=51e645e88856df580ad0de225d47b16a/B, priority=13, startTime=1679273803912; duration=0sec 2023-03-20 00:56:44,348 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:44,349 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:56:44,352 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34830 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:56:44,352 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 51e645e88856df580ad0de225d47b16a/C is initiating minor compaction (all files) 2023-03-20 00:56:44,352 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e645e88856df580ad0de225d47b16a/C in TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:44,352 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/37ebecbeb4b94e27b2659047977469af, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/5643fe50fd094b9f80cb921092b25a25, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/44af2bb3ea4549eda3b81a9bebd91e7e] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp, totalSize=34.0 K 2023-03-20 00:56:44,352 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 37ebecbeb4b94e27b2659047977469af, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=1426, earliestPutTs=1679273800139 2023-03-20 00:56:44,354 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 5643fe50fd094b9f80cb921092b25a25, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1435, earliestPutTs=1679273801259 2023-03-20 00:56:44,354 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 44af2bb3ea4549eda3b81a9bebd91e7e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1544, earliestPutTs=1679273802608 2023-03-20 00:56:44,358 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 51e645e88856df580ad0de225d47b16a/A of 51e645e88856df580ad0de225d47b16a into 876ce9e9f2c243d7bdac9856ee4ea97c(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:44,358 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:44,358 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a., storeName=51e645e88856df580ad0de225d47b16a/A, priority=13, startTime=1679273803912; duration=0sec 2023-03-20 00:56:44,358 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:44,362 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 51e645e88856df580ad0de225d47b16a#C#compaction#253 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:44,364 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=1751 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/91a6b00320ac44ea942c2d5a228c0b8a 2023-03-20 00:56:44,372 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/a7520f3c8dab4fb4b10033dac7df9d19 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/a7520f3c8dab4fb4b10033dac7df9d19 2023-03-20 00:56:44,376 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 51e645e88856df580ad0de225d47b16a/C of 51e645e88856df580ad0de225d47b16a into a7520f3c8dab4fb4b10033dac7df9d19(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:44,376 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:44,376 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a., storeName=51e645e88856df580ad0de225d47b16a/C, priority=13, startTime=1679273803912; duration=0sec 2023-03-20 00:56:44,377 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:44,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:44,384 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:44,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:44,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 394 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273864385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:44,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:44,388 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:44,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 504 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273864385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:44,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273864386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:44,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:44,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 442 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273864386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:44,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:44,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 484 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273864387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:44,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:44,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 396 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273864488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:44,490 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:44,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273864488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:44,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:44,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 486 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273864489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:44,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:44,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 398 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273864691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:44,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:44,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273864692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:44,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:44,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 488 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273864693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:44,782 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=1751 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/8fcdad622d8145d18dbcafeacd2912ee 2023-03-20 00:56:44,787 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/9d7514ef388145f6b5e4090644049052 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/9d7514ef388145f6b5e4090644049052 2023-03-20 00:56:44,792 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/9d7514ef388145f6b5e4090644049052, entries=150, sequenceid=1751, filesize=11.9 K 2023-03-20 00:56:44,793 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/91a6b00320ac44ea942c2d5a228c0b8a as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/91a6b00320ac44ea942c2d5a228c0b8a 2023-03-20 00:56:44,801 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/91a6b00320ac44ea942c2d5a228c0b8a, entries=150, sequenceid=1751, filesize=11.9 K 2023-03-20 00:56:44,802 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/8fcdad622d8145d18dbcafeacd2912ee as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/8fcdad622d8145d18dbcafeacd2912ee 2023-03-20 00:56:44,824 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/8fcdad622d8145d18dbcafeacd2912ee, entries=150, sequenceid=1751, filesize=11.9 K 2023-03-20 00:56:44,825 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~167.72 KB/171750, heapSize ~452.67 KB/463536, currentSize=20.13 KB/20610 for 51e645e88856df580ad0de225d47b16a in 883ms, sequenceid=1751, compaction requested=false 2023-03-20 00:56:44,825 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:44,825 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:44,825 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:44,825 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:44,826 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:44,826 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:44,826 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:44,827 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:44,827 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:44,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:44,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:44,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:44,827 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:44,827 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:44,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:44,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:44,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:44,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:44,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:44,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:44,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13e9effd[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:44,829 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:44,829 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:44,830 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:44,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:44,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:44,830 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:44,830 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:44,830 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:44,831 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:44,832 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:44,832 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:44,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:44,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:44,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:44,832 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:44,832 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:44,832 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:44,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:44,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:44,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:44,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:44,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:44,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:44,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:44,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:44,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:44,836 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:44,837 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:44,837 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:44,837 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:44,837 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:44,847 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:44,847 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:44,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:44,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:44,847 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:44,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:44,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:44,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:44,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:44,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:44,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:44,848 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:44,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:44,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:44,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:44,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:44,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:44,850 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:44,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:44,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:44,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:44,855 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:44,855 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:44,855 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:44,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:44,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:44,855 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:44,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:44,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:44,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:44,855 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:44,856 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:44,855 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:44,856 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:44,856 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:44,856 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:44,856 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:44,856 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:44,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:44,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:44,956 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:44,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:44,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:44,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:44,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:44,961 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:44,961 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:44,961 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:44,961 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:44,962 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:44,962 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:44,963 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:44,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:44,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:44,963 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:44,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:44,963 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:44,964 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:44,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:44,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:44,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:44,964 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:44,964 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:44,964 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:44,965 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:44,965 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:44,965 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:44,974 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:44,974 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:44,974 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:44,975 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:44,975 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51e645e88856df580ad0de225d47b16a 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-20 00:56:44,975 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=A 2023-03-20 00:56:44,975 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:44,975 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=B 2023-03-20 00:56:44,975 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:44,975 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=C 2023-03-20 00:56:44,975 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:45,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:45,004 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:45,013 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=1760 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/0f83229aa0364f27b2bfb5fd64e75e3c 2023-03-20 00:56:45,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679273805018, max timestamp=1679273805025Num uniques 859; 2023-03-20 00:56:45,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-20 00:56:45,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679273805018, max timestamp=1679273805025Num uniques 859; 2023-03-20 00:56:45,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679273805018, max timestamp=1679273805025Num uniques 859; 2023-03-20 00:56:45,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-20 00:56:45,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-20 00:56:45,040 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=1760 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/f1e228714c7c49628af618d5c8255739 2023-03-20 00:56:45,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679273805088, max timestamp=1679273805096Num uniques 1118; 2023-03-20 00:56:45,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-03-20 00:56:45,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679273805088, max timestamp=1679273805096Num uniques 1118; 2023-03-20 00:56:45,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-03-20 00:56:45,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679273805088, max timestamp=1679273805096Num uniques 1118; 2023-03-20 00:56:45,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-03-20 00:56:45,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679273805126, max timestamp=1679273805134Num uniques 1077; 2023-03-20 00:56:45,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-03-20 00:56:45,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679273805126, max timestamp=1679273805134Num uniques 1077; 2023-03-20 00:56:45,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-03-20 00:56:45,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679273805126, max timestamp=1679273805134Num uniques 1077; 2023-03-20 00:56:45,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-03-20 00:56:45,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679273805151, max timestamp=1679273805165Num uniques 1086; 2023-03-20 00:56:45,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-03-20 00:56:45,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679273805151, max timestamp=1679273805165Num uniques 1086; 2023-03-20 00:56:45,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-03-20 00:56:45,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679273805151, max timestamp=1679273805165Num uniques 1086; 2023-03-20 00:56:45,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-03-20 00:56:45,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679273805178, max timestamp=1679273805190Num uniques 1095; 2023-03-20 00:56:45,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679273805178, max timestamp=1679273805190Num uniques 1095; 2023-03-20 00:56:45,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-03-20 00:56:45,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-03-20 00:56:45,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679273805178, max timestamp=1679273805190Num uniques 1095; 2023-03-20 00:56:45,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-03-20 00:56:45,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679273805204, max timestamp=1679273805212Num uniques 1054; 2023-03-20 00:56:45,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679273805204, max timestamp=1679273805212Num uniques 1054; 2023-03-20 00:56:45,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-20 00:56:45,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-20 00:56:45,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679273805204, max timestamp=1679273805212Num uniques 1054; 2023-03-20 00:56:45,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-20 00:56:45,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679273805236, max timestamp=1679273805243Num uniques 1013; 2023-03-20 00:56:45,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-03-20 00:56:45,247 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679273805236, max timestamp=1679273805243Num uniques 1013; 2023-03-20 00:56:45,247 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=6870, suffix heap size=133320,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=150 2023-03-20 00:56:45,247 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679273805236, max timestamp=1679273805243Num uniques 1013; 2023-03-20 00:56:45,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=6870, suffix heap size=133320,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=150 2023-03-20 00:56:45,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679273805256, max timestamp=1679273805271Num uniques 1072; 2023-03-20 00:56:45,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-20 00:56:45,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679273805256, max timestamp=1679273805271Num uniques 1009; 2023-03-20 00:56:45,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=7872, suffix heap size=132904,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=172 2023-03-20 00:56:45,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679273805256, max timestamp=1679273805271Num uniques 1009; 2023-03-20 00:56:45,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=7872, suffix heap size=132904,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=172 2023-03-20 00:56:45,302 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679273805295, max timestamp=1679273805300Num uniques 1031; 2023-03-20 00:56:45,302 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679273805295, max timestamp=1679273805300Num uniques 1081; 2023-03-20 00:56:45,302 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-03-20 00:56:45,302 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-03-20 00:56:45,302 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679273805295, max timestamp=1679273805300Num uniques 1031; 2023-03-20 00:56:45,302 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-03-20 00:56:45,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1679273805318, max timestamp=1679273805324Num uniques 1040; 2023-03-20 00:56:45,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-03-20 00:56:45,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679273805318, max timestamp=1679273805324Num uniques 1040; 2023-03-20 00:56:45,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679273805318, max timestamp=1679273805324Num uniques 1040; 2023-03-20 00:56:45,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-03-20 00:56:45,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-03-20 00:56:45,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679273805347, max timestamp=1679273805353Num uniques 1049; 2023-03-20 00:56:45,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-03-20 00:56:45,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679273805347, max timestamp=1679273805353Num uniques 1099; 2023-03-20 00:56:45,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-03-20 00:56:45,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679273805347, max timestamp=1679273805353Num uniques 1049; 2023-03-20 00:56:45,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-03-20 00:56:45,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1679273805371, max timestamp=1679273805381Num uniques 1058; 2023-03-20 00:56:45,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=11810, suffix heap size=138000,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=258 2023-03-20 00:56:45,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1679273805373, max timestamp=1679273805381Num uniques 1058; 2023-03-20 00:56:45,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1679273805373, max timestamp=1679273805381Num uniques 1058; 2023-03-20 00:56:45,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-03-20 00:56:45,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-03-20 00:56:45,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=317, cellSize=14512, totalHeapSize=33208, min timestamp=1679273805394, max timestamp=1679273805402Num uniques 1017; 2023-03-20 00:56:45,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679273805398, max timestamp=1679273805402Num uniques 1017; 2023-03-20 00:56:45,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=14512, suffix heap size=133736,new segment heap size=33208 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=317 2023-03-20 00:56:45,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=317, cellSize=14512, totalHeapSize=33208, min timestamp=1679273805394, max timestamp=1679273805402Num uniques 1117; 2023-03-20 00:56:45,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-03-20 00:56:45,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=14512, suffix heap size=144136,new segment heap size=33208 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=317 2023-03-20 00:56:45,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1679273805411, max timestamp=1679273805423Num uniques 1176; 2023-03-20 00:56:45,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=53856, new segment data size=10346, suffix heap size=150272,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1176, new segment cells count=226 2023-03-20 00:56:45,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1679273805411, max timestamp=1679273805423Num uniques 1176; 2023-03-20 00:56:45,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1679273805411, max timestamp=1679273805423Num uniques 1026; 2023-03-20 00:56:45,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=53856, new segment data size=10346, suffix heap size=150272,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1176, new segment cells count=226 2023-03-20 00:56:45,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=10346, suffix heap size=134672,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=226 2023-03-20 00:56:45,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1679273805435, max timestamp=1679273805442Num uniques 1085; 2023-03-20 00:56:45,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=15340, suffix heap size=140808,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=335 2023-03-20 00:56:45,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1679273805435, max timestamp=1679273805442Num uniques 1085; 2023-03-20 00:56:45,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=15340, suffix heap size=140808,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=335 2023-03-20 00:56:45,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1679273805435, max timestamp=1679273805442Num uniques 1085; 2023-03-20 00:56:45,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=15340, suffix heap size=140808,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=335 2023-03-20 00:56:45,468 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=1760 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/00722440e933440ab6dc3b0fb764ddea 2023-03-20 00:56:45,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1679273805465, max timestamp=1679273805471Num uniques 1194; 2023-03-20 00:56:45,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1679273805465, max timestamp=1679273805471Num uniques 1194; 2023-03-20 00:56:45,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=54684, new segment data size=11174, suffix heap size=152144,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1194, new segment cells count=244 2023-03-20 00:56:45,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=54684, new segment data size=11174, suffix heap size=152144,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1194, new segment cells count=244 2023-03-20 00:56:45,475 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/0f83229aa0364f27b2bfb5fd64e75e3c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/0f83229aa0364f27b2bfb5fd64e75e3c 2023-03-20 00:56:45,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679273805465, max timestamp=1679273805471Num uniques 1194; 2023-03-20 00:56:45,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=54684, new segment data size=6870, suffix heap size=152144,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1194, new segment cells count=150 2023-03-20 00:56:45,479 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/0f83229aa0364f27b2bfb5fd64e75e3c, entries=150, sequenceid=1760, filesize=11.9 K 2023-03-20 00:56:45,480 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/f1e228714c7c49628af618d5c8255739 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/f1e228714c7c49628af618d5c8255739 2023-03-20 00:56:45,485 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/f1e228714c7c49628af618d5c8255739, entries=150, sequenceid=1760, filesize=11.9 K 2023-03-20 00:56:45,486 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/00722440e933440ab6dc3b0fb764ddea as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/00722440e933440ab6dc3b0fb764ddea 2023-03-20 00:56:45,494 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/00722440e933440ab6dc3b0fb764ddea, entries=150, sequenceid=1760, filesize=11.9 K 2023-03-20 00:56:45,494 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-03-20 00:56:45,495 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=103.14 KB/105616 for 51e645e88856df580ad0de225d47b16a in 520ms, sequenceid=1760, compaction requested=true 2023-03-20 00:56:45,495 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:45,495 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:45,496 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:45,496 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:45,496 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,496 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:45,496 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:45,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:45,497 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51e645e88856df580ad0de225d47b16a 3/3 column families, dataSize=116.56 KB heapSize=327.63 KB 2023-03-20 00:56:45,497 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=A 2023-03-20 00:56:45,497 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,497 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:56:45,497 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=B 2023-03-20 00:56:45,497 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:56:45,497 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=C 2023-03-20 00:56:45,497 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,497 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:45,497 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:56:45,497 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:45,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:45,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:45,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:45,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:45,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:45,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:45,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4880e813[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:45,499 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:45,499 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,500 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,500 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:45,500 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:45,500 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:45,501 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:45,501 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:45,501 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:45,502 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,502 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:45,503 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:45,503 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:45,503 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:45,503 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:45,503 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:45,503 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:45,504 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,504 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:45,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,504 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:45,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:45,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:45,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:45,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,505 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:45,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:45,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:45,506 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:45,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:45,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:45,509 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,509 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:45,509 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:45,509 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:45,509 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:45,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:45,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:45,509 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:45,509 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,509 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,509 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,509 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:45,509 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:45,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:45,510 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:45,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:45,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:45,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:45,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:45,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 455 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273865509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:45,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 580 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273865510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:45,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 487 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273865512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:45,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 523 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273865512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:45,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 461 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273865518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,610 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:45,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:45,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:45,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:45,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:45,613 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:45,613 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:45,613 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:45,613 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:45,614 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,614 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:45,615 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:45,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:45,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:45,615 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:45,615 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,615 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:45,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:45,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:45,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:45,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:45,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 457 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273865618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:45,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 582 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273865618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:45,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 525 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273865619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,621 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:45,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 489 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273865619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,624 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:45,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 463 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273865623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,630 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:45,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:45,630 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:45,630 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:45,630 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:45,638 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:45,638 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:45,638 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:45,638 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:45,638 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:45,638 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:45,638 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:45,638 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:45,638 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,638 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:45,638 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:45,639 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,639 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:45,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:45,640 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,640 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:45,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:45,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:45,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:45,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:45,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:45,641 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:45,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f990edb[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:45,641 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,642 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,642 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:45,642 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:45,642 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:45,642 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,642 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:45,644 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,644 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:45,644 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:45,644 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:45,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:45,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:45,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:45,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:45,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:45,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:45,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:45,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:45,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:45,646 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:45,646 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:45,646 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:45,646 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:45,646 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:45,648 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,648 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,648 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:45,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:45,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:45,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:45,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:45,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:45,648 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:45,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:45,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:45,650 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:45,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:45,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,658 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:45,658 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:45,658 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:45,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:45,658 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:45,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:45,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:45,658 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:45,659 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:45,659 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:45,659 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:45,659 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,659 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,659 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:45,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:45,659 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:45,759 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:45,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:45,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:45,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:45,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:45,762 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:45,762 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:45,762 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:45,762 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:45,762 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,762 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:45,763 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:45,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:45,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:45,763 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:45,763 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,763 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:45,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:45,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:45,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:45,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:45,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:45,764 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:45,764 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:45,764 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:45,764 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:45,764 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:45,764 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:45,764 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:45,764 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:45,764 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:45,764 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:45,764 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:45,765 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,765 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:45,765 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:45,765 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,765 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:45,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:45,766 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,766 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:45,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:45,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:45,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:45,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:45,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:45,767 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:45,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25e789c5[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:45,767 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,768 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,768 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:45,768 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:45,768 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:45,769 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:45,769 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:45,769 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:45,769 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,769 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:45,770 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:45,770 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:45,770 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:45,770 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:45,770 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:45,770 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:45,771 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,771 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:45,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,771 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:45,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:45,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:45,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:45,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:45,771 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:45,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:45,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:45,772 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:45,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:45,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,774 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:45,775 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:45,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:45,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:45,775 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:45,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:45,775 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:45,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:45,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:45,775 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:45,775 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:45,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:45,775 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:45,775 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:45,775 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,775 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,775 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,775 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,822 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:45,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 459 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273865821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:45,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 584 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273865821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:45,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 527 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273865821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:45,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 491 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273865823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:45,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 465 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273865827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,875 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:45,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:45,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:45,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:45,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:45,878 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:45,878 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:45,878 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:45,878 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:45,879 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,879 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:45,880 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:45,880 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:45,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:45,880 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,880 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:45,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:45,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:45,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:45,881 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:45,881 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:45,881 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:45,881 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:45,881 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:45,881 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:45,881 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:45,881 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:45,881 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:45,881 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:45,881 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:45,881 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:45,881 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:45,881 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:45,881 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,881 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:45,882 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:45,882 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,882 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:45,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:45,883 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,883 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:45,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:45,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:45,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:45,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:45,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:45,884 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:45,885 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79ad1c83[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:45,885 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,885 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:45,885 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:45,885 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:45,885 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,885 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:45,886 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,886 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:45,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:45,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:45,886 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:45,886 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:45,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:45,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:45,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:45,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:45,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:45,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:45,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:45,889 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:45,889 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:45,889 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:45,889 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:45,889 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:45,890 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,890 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:45,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:45,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,890 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:45,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:45,890 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:45,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:45,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:45,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:45,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:45,892 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:45,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:45,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,894 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,894 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:45,894 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:45,894 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:45,894 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:45,894 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,894 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:45,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:45,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:45,894 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:45,895 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,894 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:45,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:45,895 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:45,895 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:45,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:45,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:45,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:45,913 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=2051 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/b5ecbdac93db43108a28754ab6030523 2023-03-20 00:56:45,995 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:45,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:45,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:45,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:45,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:45,998 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:45,998 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:45,998 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:45,998 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:45,999 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:45,999 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:45,999 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:45,999 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:45,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:45,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:46,000 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,000 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:46,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:46,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,000 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:46,001 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:46,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:46,001 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:46,001 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:46,001 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:46,001 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:46,001 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:46,001 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:46,001 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:46,001 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:46,001 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:46,001 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:46,001 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:46,001 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,001 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:46,001 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:46,003 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:46,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:46,003 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:46,004 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,004 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:46,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:46,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36a2b3d5[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:46,005 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:46,005 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,006 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,006 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:46,006 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,006 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:46,006 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:46,006 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:46,007 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:46,007 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,007 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:46,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:46,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:46,007 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:46,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:46,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:46,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:46,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:46,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:46,009 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:46,009 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:46,010 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:46,010 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:46,010 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:46,010 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,010 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:46,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:46,011 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:46,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:46,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:46,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:46,011 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,012 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,014 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,014 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,014 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,014 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,014 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,014 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:46,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:46,014 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,014 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:46,014 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:46,014 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:46,015 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:46,015 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:46,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:46,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:46,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:46,115 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:46,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:46,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:46,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:46,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:46,118 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:46,118 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:46,118 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:46,118 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:46,118 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,118 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,119 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:46,119 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:46,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:46,120 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,120 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:46,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:46,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:46,121 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:46,121 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:46,121 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:46,121 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:46,121 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:46,121 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:46,121 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:46,121 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:46,121 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:46,122 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:46,122 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:46,122 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,122 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:46,122 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:46,123 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,123 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:46,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:46,123 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,123 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:46,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:46,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:46,125 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:46,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@512a8c0c[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:46,125 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,126 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,126 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:46,126 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:46,126 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:46,126 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,126 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:46,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:46,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 461 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273866123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,127 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:46,127 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:46,127 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:46,127 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:46,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:46,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:46,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:46,128 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:46,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 529 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273866125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,128 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:46,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 586 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273866125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,129 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:46,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 493 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273866127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:46,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:46,130 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:46,130 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:46,130 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:46,130 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:46,130 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:46,131 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,131 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:46,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,131 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:46,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:46,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:46,132 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:46,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,133 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,134 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:46,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 467 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273866131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,135 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:46,135 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:46,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:46,135 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,135 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,135 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,135 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,135 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:46,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:46,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:46,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:46,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:46,135 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,135 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:46,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:46,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:46,135 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,135 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:46,136 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,236 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:46,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:46,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:46,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:46,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:46,241 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:46,242 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:46,242 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:46,242 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:46,242 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,242 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,243 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:46,243 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:46,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:46,244 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,244 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:46,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:46,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:46,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:46,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:46,246 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:46,246 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:46,246 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:46,246 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:46,246 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:46,246 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:46,246 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:46,246 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:46,246 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:46,246 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:46,246 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:46,246 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,246 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:46,246 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:46,247 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:46,247 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:46,248 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,248 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:46,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:46,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:46,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@489a8e4d[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:46,251 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:46,251 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,252 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,252 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:46,252 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:46,252 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:46,257 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:46,258 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:46,258 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:46,260 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,262 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:46,262 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:46,262 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:46,262 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:46,262 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:46,262 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:46,262 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:46,268 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,268 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:46,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,268 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:46,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:46,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:46,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:46,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:46,269 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,272 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:46,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,280 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:46,280 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:46,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:46,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:46,280 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,280 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,280 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,280 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:46,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:46,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:46,280 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,281 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,281 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,281 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,281 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:46,281 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:46,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:46,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:46,340 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=2051 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/d6f00b87da8f40b49880cdbe89701bfa 2023-03-20 00:56:46,381 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:46,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:46,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:46,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:46,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:46,406 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:46,406 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:46,406 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:46,406 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:46,414 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,414 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,415 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,415 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:46,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:46,415 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,415 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:46,426 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:46,426 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:46,426 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:46,426 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:46,426 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:46,427 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:46,427 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:46,427 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:46,427 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:46,427 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:46,427 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:46,427 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:46,427 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:46,427 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,427 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:46,427 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:46,430 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,430 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:46,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:46,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:46,431 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,431 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:46,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:46,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:46,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:46,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5942cad9[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:46,432 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:46,432 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,433 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:46,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:46,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:46,434 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,435 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:46,435 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:46,435 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,435 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:46,435 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:46,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:46,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:46,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:46,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:46,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:46,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:46,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:46,437 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:46,437 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:46,437 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:46,437 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:46,437 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:46,438 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,438 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,438 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:46,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:46,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:46,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:46,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:46,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:46,439 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,440 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:46,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,442 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:46,442 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:46,442 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:46,442 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:46,442 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:46,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:46,442 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,442 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,442 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,442 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,442 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:46,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:46,443 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:46,443 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:46,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:46,543 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:46,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:46,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:46,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:46,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:46,547 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:46,547 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:46,547 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:46,547 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:46,547 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,548 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,548 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:46,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:46,548 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:46,549 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,549 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:46,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:46,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,550 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:46,550 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:46,550 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:46,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:46,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:46,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:46,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:46,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:46,550 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:46,550 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:46,551 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:46,551 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:46,551 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:46,551 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:46,551 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,551 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:46,551 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:46,552 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,552 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:46,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:46,552 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,552 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:46,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:46,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:46,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:46,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:46,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a5df63f[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:46,553 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:46,554 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,554 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,554 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:46,554 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:46,554 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:46,554 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,554 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:46,555 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,555 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:46,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:46,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:46,555 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:46,555 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:46,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:46,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:46,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:46,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:46,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:46,557 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:46,557 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:46,557 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:46,557 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:46,557 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:46,558 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,558 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:46,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,558 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:46,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:46,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:46,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:46,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:46,559 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:46,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,561 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:46,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,562 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,562 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,562 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,562 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:46,562 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:46,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:46,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:46,562 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:46,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:46,562 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:46,562 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,562 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:46,562 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,563 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:46,563 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:46,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:46,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:46,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 531 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48368 deadline: 1679273866629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:46,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 588 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48408 deadline: 1679273866629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:46,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 463 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48410 deadline: 1679273866630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:46,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 495 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48422 deadline: 1679273866634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,640 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:46,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 469 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48428 deadline: 1679273866638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,663 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:46,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:46,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:46,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:46,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:46,665 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:46,665 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:46,665 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:46,665 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:46,666 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,666 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,666 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,666 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:46,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:46,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:46,667 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,667 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:46,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:46,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:46,668 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:46,668 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:46,668 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:46,668 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:46,668 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:46,668 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:46,668 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:46,668 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:46,668 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:46,668 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. as already flushing 2023-03-20 00:56:46,668 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:46,668 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:46,668 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:46,668 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,669 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:46,669 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:46,669 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,669 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:46,670 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,670 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:46,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:46,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:46,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:46,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:46,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f124f94[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:46,671 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:46,671 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,671 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,671 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:46,671 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:46,671 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:46,671 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,671 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:46,672 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,672 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:46,672 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:46,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:46,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:46,672 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:46,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:46,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:46,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:46,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:46,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:46,674 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:46,674 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:46,675 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:46,675 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:46,675 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:46,676 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,676 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:46,676 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:46,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:46,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:46,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:46,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:46,683 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:46,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,685 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:46,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:46,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,688 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,688 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:46,688 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:46,688 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:46,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:46,688 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:46,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:46,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:46,689 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:46,688 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,689 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:46,689 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:46,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:46,689 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,689 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:46,689 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,749 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=36.05 KB at sequenceid=2051 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/a1827b019d3a49cca39993d941ed036a 2023-03-20 00:56:46,754 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/b5ecbdac93db43108a28754ab6030523 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/b5ecbdac93db43108a28754ab6030523 2023-03-20 00:56:46,758 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/b5ecbdac93db43108a28754ab6030523, entries=150, sequenceid=2051, filesize=11.9 K 2023-03-20 00:56:46,761 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/d6f00b87da8f40b49880cdbe89701bfa as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/d6f00b87da8f40b49880cdbe89701bfa 2023-03-20 00:56:46,768 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/d6f00b87da8f40b49880cdbe89701bfa, entries=150, sequenceid=2051, filesize=11.9 K 2023-03-20 00:56:46,769 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/a1827b019d3a49cca39993d941ed036a as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/a1827b019d3a49cca39993d941ed036a 2023-03-20 00:56:46,782 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/a1827b019d3a49cca39993d941ed036a, entries=150, sequenceid=2051, filesize=11.9 K 2023-03-20 00:56:46,783 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~116.56 KB/119356, heapSize ~327.58 KB/335440, currentSize=73.80 KB/75570 for 51e645e88856df580ad0de225d47b16a in 1286ms, sequenceid=2051, compaction requested=true 2023-03-20 00:56:46,783 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:46,783 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:46,783 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 00:56:46,783 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 00:56:46,783 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:46,783 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:46,784 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 00:56:46,784 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 00:56:46,784 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 51e645e88856df580ad0de225d47b16a/A is initiating minor compaction (all files) 2023-03-20 00:56:46,785 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 51e645e88856df580ad0de225d47b16a/B is initiating minor compaction (all files) 2023-03-20 00:56:46,785 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e645e88856df580ad0de225d47b16a/A in TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:46,785 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/876ce9e9f2c243d7bdac9856ee4ea97c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/9d7514ef388145f6b5e4090644049052, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/0f83229aa0364f27b2bfb5fd64e75e3c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/b5ecbdac93db43108a28754ab6030523] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp, totalSize=48.4 K 2023-03-20 00:56:46,785 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e645e88856df580ad0de225d47b16a/B in TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:46,785 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/e30fd4dcecc84f39bd3e717dc3b4f8fe, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/91a6b00320ac44ea942c2d5a228c0b8a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/f1e228714c7c49628af618d5c8255739, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/d6f00b87da8f40b49880cdbe89701bfa] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp, totalSize=48.4 K 2023-03-20 00:56:46,785 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 876ce9e9f2c243d7bdac9856ee4ea97c, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=1544, earliestPutTs=1679273802608 2023-03-20 00:56:46,785 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting e30fd4dcecc84f39bd3e717dc3b4f8fe, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=1544, earliestPutTs=1679273802608 2023-03-20 00:56:46,785 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 9d7514ef388145f6b5e4090644049052, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1751, earliestPutTs=1679273803250 2023-03-20 00:56:46,789 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:46,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:46,790 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 0f83229aa0364f27b2bfb5fd64e75e3c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1760, earliestPutTs=1679273804383 2023-03-20 00:56:46,790 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 91a6b00320ac44ea942c2d5a228c0b8a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1751, earliestPutTs=1679273803250 2023-03-20 00:56:46,791 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting b5ecbdac93db43108a28754ab6030523, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=2051, earliestPutTs=1679273805482 2023-03-20 00:56:46,791 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting f1e228714c7c49628af618d5c8255739, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1760, earliestPutTs=1679273804383 2023-03-20 00:56:46,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:46,792 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting d6f00b87da8f40b49880cdbe89701bfa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=2051, earliestPutTs=1679273805482 2023-03-20 00:56:46,814 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 51e645e88856df580ad0de225d47b16a#B#compaction#261 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:46,818 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51e645e88856df580ad0de225d47b16a#A#compaction#262 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:46,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:46,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:46,822 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:46,822 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:46,822 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:46,822 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:46,823 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,823 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,823 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:46,823 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:46,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:46,824 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:46,824 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:46,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:46,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:46,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:46,825 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:46,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:46,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:46,838 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:46,838 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:46,838 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:46,853 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:46,853 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:46,853 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:46,853 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. started... 2023-03-20 00:56:46,853 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51e645e88856df580ad0de225d47b16a 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-20 00:56:46,859 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=A 2023-03-20 00:56:46,859 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:46,859 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=B 2023-03-20 00:56:46,859 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:46,859 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=C 2023-03-20 00:56:46,859 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:46,950 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=2065 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/a6724d4463824f6aaa9e7aa131e124dc 2023-03-20 00:56:47,292 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/ff13d52fae21495a9c7a709ea6219a76 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/ff13d52fae21495a9c7a709ea6219a76 2023-03-20 00:56:47,298 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 51e645e88856df580ad0de225d47b16a/B of 51e645e88856df580ad0de225d47b16a into ff13d52fae21495a9c7a709ea6219a76(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:47,298 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:47,298 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a., storeName=51e645e88856df580ad0de225d47b16a/B, priority=12, startTime=1679273806783; duration=0sec 2023-03-20 00:56:47,298 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:47,298 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 00:56:47,299 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 00:56:47,299 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 51e645e88856df580ad0de225d47b16a/C is initiating minor compaction (all files) 2023-03-20 00:56:47,300 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51e645e88856df580ad0de225d47b16a/C in TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:47,300 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/a7520f3c8dab4fb4b10033dac7df9d19, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/8fcdad622d8145d18dbcafeacd2912ee, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/00722440e933440ab6dc3b0fb764ddea, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/a1827b019d3a49cca39993d941ed036a] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp, totalSize=48.4 K 2023-03-20 00:56:47,300 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting a7520f3c8dab4fb4b10033dac7df9d19, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=1544, earliestPutTs=1679273802608 2023-03-20 00:56:47,300 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 8fcdad622d8145d18dbcafeacd2912ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1751, earliestPutTs=1679273803250 2023-03-20 00:56:47,301 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 00722440e933440ab6dc3b0fb764ddea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1760, earliestPutTs=1679273804383 2023-03-20 00:56:47,301 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting a1827b019d3a49cca39993d941ed036a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=2051, earliestPutTs=1679273805482 2023-03-20 00:56:47,314 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/b1b6323acdbb4dfab73fe9c0a4b9b6e0 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/b1b6323acdbb4dfab73fe9c0a4b9b6e0 2023-03-20 00:56:47,316 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 51e645e88856df580ad0de225d47b16a#C#compaction#265 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:47,320 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 51e645e88856df580ad0de225d47b16a/A of 51e645e88856df580ad0de225d47b16a into b1b6323acdbb4dfab73fe9c0a4b9b6e0(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:47,320 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:47,320 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a., storeName=51e645e88856df580ad0de225d47b16a/A, priority=12, startTime=1679273806783; duration=0sec 2023-03-20 00:56:47,320 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:47,347 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/a2dc2ad6ab5f480caa822206b69c3d58 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/a2dc2ad6ab5f480caa822206b69c3d58 2023-03-20 00:56:47,351 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 51e645e88856df580ad0de225d47b16a/C of 51e645e88856df580ad0de225d47b16a into a2dc2ad6ab5f480caa822206b69c3d58(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:47,352 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:47,352 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a., storeName=51e645e88856df580ad0de225d47b16a/C, priority=12, startTime=1679273806783; duration=0sec 2023-03-20 00:56:47,352 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:47,384 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=2065 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/6f4f832204c94b85b42b4fbd38bc7c5f 2023-03-20 00:56:47,396 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=2065 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/9ce03f4d66a94f529bc6f8a9f8124e64 2023-03-20 00:56:47,400 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/a6724d4463824f6aaa9e7aa131e124dc as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/a6724d4463824f6aaa9e7aa131e124dc 2023-03-20 00:56:47,404 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/a6724d4463824f6aaa9e7aa131e124dc, entries=100, sequenceid=2065, filesize=9.5 K 2023-03-20 00:56:47,405 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/6f4f832204c94b85b42b4fbd38bc7c5f as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/6f4f832204c94b85b42b4fbd38bc7c5f 2023-03-20 00:56:47,409 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/6f4f832204c94b85b42b4fbd38bc7c5f, entries=100, sequenceid=2065, filesize=9.5 K 2023-03-20 00:56:47,410 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/9ce03f4d66a94f529bc6f8a9f8124e64 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/9ce03f4d66a94f529bc6f8a9f8124e64 2023-03-20 00:56:47,414 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/9ce03f4d66a94f529bc6f8a9f8124e64, entries=100, sequenceid=2065, filesize=9.5 K 2023-03-20 00:56:47,415 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 51e645e88856df580ad0de225d47b16a in 562ms, sequenceid=2065, compaction requested=false 2023-03-20 00:56:47,415 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:47,415 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:47,415 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:47,415 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:47,415 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:47,415 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:47,415 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:47,416 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:47,416 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:47,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:47,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:47,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:47,416 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:47,417 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:47,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:47,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:47,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:47,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:47,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:47,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:47,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63665edd[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:47,418 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:47,418 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:47,419 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:47,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:47,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:47,419 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:47,419 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:47,419 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:47,419 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:47,419 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:47,421 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:47,421 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:47,421 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:47,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:47,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:47,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:47,421 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:47,428 DEBUG [Thread-1430] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4cbc74e7 to 127.0.0.1:60547 2023-03-20 00:56:47,428 DEBUG [Thread-1434] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d22a55e to 127.0.0.1:60547 2023-03-20 00:56:47,428 DEBUG [Thread-1430] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:56:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:47,436 DEBUG [Thread-1434] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:56:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:47,436 DEBUG [Thread-1432] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3c5d1710 to 127.0.0.1:60547 2023-03-20 00:56:47,436 DEBUG [Thread-1428] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ade356f to 127.0.0.1:60547 2023-03-20 00:56:47,437 DEBUG [Thread-1432] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:56:47,437 DEBUG [Thread-1428] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:56:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:47,438 DEBUG [Thread-1436] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2179ead9 to 127.0.0.1:60547 2023-03-20 00:56:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:47,438 DEBUG [Thread-1436] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:56:47,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:47,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:47,439 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:47,439 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:47,439 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:47,439 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:47,439 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:47,440 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:47,440 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:47,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:47,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:47,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:47,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:47,440 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:47,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:47,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:47,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:47,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:47,440 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:47,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:47,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:47,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:47,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:47,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:47,441 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:47,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:47,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:47,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:47,443 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:47,443 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:47,443 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:47,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:47,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:47,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:47,443 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:47,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:47,443 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:47,443 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:47,443 DEBUG [Thread-1427] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:47,443 DEBUG [Thread-1427] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:47,443 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:47,443 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:47,443 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:47,443 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:47,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:47,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:47,543 DEBUG [Thread-1427] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:47,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:47,641 DEBUG [Thread-1417] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c466b6c to 127.0.0.1:60547 2023-03-20 00:56:47,641 DEBUG [Thread-1419] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x46d74611 to 127.0.0.1:60547 2023-03-20 00:56:47,641 DEBUG [Thread-1423] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a8a575d to 127.0.0.1:60547 2023-03-20 00:56:47,642 DEBUG [Thread-1423] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:56:47,641 DEBUG [Thread-1417] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:56:47,641 DEBUG [Thread-1419] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:56:47,646 DEBUG [Thread-1421] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3f837278 to 127.0.0.1:60547 2023-03-20 00:56:47,646 DEBUG [Thread-1421] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:56:47,647 DEBUG [Thread-1425] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6347f2e6 to 127.0.0.1:60547 2023-03-20 00:56:47,647 DEBUG [Thread-1425] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:56:47,647 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-20 00:56:47,647 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(394): wrote 386 2023-03-20 00:56:47,647 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(394): wrote 465 2023-03-20 00:56:47,647 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(394): wrote 350 2023-03-20 00:56:47,647 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(394): wrote 414 2023-03-20 00:56:47,647 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(394): wrote 366 2023-03-20 00:56:47,647 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-20 00:56:47,647 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(398): read 5078 2023-03-20 00:56:47,648 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(398): read 4667 2023-03-20 00:56:47,648 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(398): read 4573 2023-03-20 00:56:47,648 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(398): read 5050 2023-03-20 00:56:47,648 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(398): read 4687 2023-03-20 00:56:47,648 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-20 00:56:47,648 INFO [Listener at localhost.localdomain/43361] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-03-20 00:56:47,648 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7117e4df to 127.0.0.1:60547 2023-03-20 00:56:47,648 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:56:47,649 INFO [Listener at localhost.localdomain/43361] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-20 00:56:47,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-03-20 00:56:47,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-20 00:56:47,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-20 00:56:47,655 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679273807654"}]},"ts":"1679273807654"} 2023-03-20 00:56:47,655 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-20 00:56:47,747 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-20 00:56:47,748 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=51e645e88856df580ad0de225d47b16a, UNASSIGN}] 2023-03-20 00:56:47,750 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=51e645e88856df580ad0de225d47b16a, UNASSIGN 2023-03-20 00:56:47,750 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=51e645e88856df580ad0de225d47b16a, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:47,751 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273807750"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679273807750"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679273807750"}]},"ts":"1679273807750"} 2023-03-20 00:56:47,752 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199}] 2023-03-20 00:56:47,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-20 00:56:47,904 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:47,904 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 51e645e88856df580ad0de225d47b16a, disabling compactions & flushes 2023-03-20 00:56:47,904 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:47,904 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:47,904 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. after waiting 0 ms 2023-03-20 00:56:47,904 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:47,904 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 51e645e88856df580ad0de225d47b16a 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-20 00:56:47,904 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=A 2023-03-20 00:56:47,904 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:47,905 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=B 2023-03-20 00:56:47,905 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:47,905 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51e645e88856df580ad0de225d47b16a, store=C 2023-03-20 00:56:47,905 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:47,912 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=2076 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/4886b40ac25b4eff9aab97a7b649cf36 2023-03-20 00:56:47,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-20 00:56:48,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-20 00:56:48,326 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=2076 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/985354926c7f4f16bcf83c40468dd36f 2023-03-20 00:56:48,339 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=2076 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/add7825368334f3eab14395829b96c4b 2023-03-20 00:56:48,345 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/A/4886b40ac25b4eff9aab97a7b649cf36 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/4886b40ac25b4eff9aab97a7b649cf36 2023-03-20 00:56:48,350 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/4886b40ac25b4eff9aab97a7b649cf36, entries=100, sequenceid=2076, filesize=9.5 K 2023-03-20 00:56:48,351 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/B/985354926c7f4f16bcf83c40468dd36f as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/985354926c7f4f16bcf83c40468dd36f 2023-03-20 00:56:48,356 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/985354926c7f4f16bcf83c40468dd36f, entries=100, sequenceid=2076, filesize=9.5 K 2023-03-20 00:56:48,357 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/.tmp/C/add7825368334f3eab14395829b96c4b as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/add7825368334f3eab14395829b96c4b 2023-03-20 00:56:48,363 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/add7825368334f3eab14395829b96c4b, entries=100, sequenceid=2076, filesize=9.5 K 2023-03-20 00:56:48,364 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 51e645e88856df580ad0de225d47b16a in 460ms, sequenceid=2076, compaction requested=true 2023-03-20 00:56:48,391 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/b96464e5dfc8454190814d7d4215e006, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/90d9c010a0d943d8abeb27a95fbce5c2, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/c8ae9be0779c45de968ed2e4feef9081, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/45b5dd34403e43a684e66863b8cafd28, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/1b2b0a7eca38407ea9d4f01b9c30ff87, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/409496aa6b024011963b35c34266a476, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/c0939baa4fa64dd3b5e720389e45128c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/792dca22532f49fd81bacc7a4b895bc6, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/f8373e59a336466ea90c2c02c076b395, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/7e78a26f5bde488bac895b60289e8b12, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/b924b94e8117445e80a133f74ffa7494, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/382070e20f13462d8dc2280f17ba5500, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/dab575a7e3d44b4cb5f4a5ffb1541c6c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/49f179b5719b4df8b44eebf64d7209f5, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/075c8903933e4cf8808edb1c88d3ded3, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/1cf01d060eb541d2bc782ee67422c6c8, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/8aa358ef68184298ac3c2d4ebd5a568a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/aef15723df944c42928614a1168dfdb4, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/526e47eccbc341a0a50889458a9f0419, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/88ba241b942d42d7aa7fdb20121c8364, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/4d9f0c21a9a043029a9c02db2154dccf, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/e5a7148e11e54c8da91e9a9161ae7ee5, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/aaa3277230034f33b9035b2fe37703f7, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/876ce9e9f2c243d7bdac9856ee4ea97c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/5b2c6ba1546f4dcc92447a98b8f4fecb, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/9d7514ef388145f6b5e4090644049052, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/0f83229aa0364f27b2bfb5fd64e75e3c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/b5ecbdac93db43108a28754ab6030523] to archive 2023-03-20 00:56:48,400 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 00:56:48,402 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/b96464e5dfc8454190814d7d4215e006 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/b96464e5dfc8454190814d7d4215e006 2023-03-20 00:56:48,403 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/90d9c010a0d943d8abeb27a95fbce5c2 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/90d9c010a0d943d8abeb27a95fbce5c2 2023-03-20 00:56:48,405 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/c8ae9be0779c45de968ed2e4feef9081 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/c8ae9be0779c45de968ed2e4feef9081 2023-03-20 00:56:48,406 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/45b5dd34403e43a684e66863b8cafd28 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/45b5dd34403e43a684e66863b8cafd28 2023-03-20 00:56:48,407 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/1b2b0a7eca38407ea9d4f01b9c30ff87 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/1b2b0a7eca38407ea9d4f01b9c30ff87 2023-03-20 00:56:48,408 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/409496aa6b024011963b35c34266a476 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/409496aa6b024011963b35c34266a476 2023-03-20 00:56:48,410 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/c0939baa4fa64dd3b5e720389e45128c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/c0939baa4fa64dd3b5e720389e45128c 2023-03-20 00:56:48,414 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/792dca22532f49fd81bacc7a4b895bc6 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/792dca22532f49fd81bacc7a4b895bc6 2023-03-20 00:56:48,416 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/f8373e59a336466ea90c2c02c076b395 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/f8373e59a336466ea90c2c02c076b395 2023-03-20 00:56:48,417 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/7e78a26f5bde488bac895b60289e8b12 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/7e78a26f5bde488bac895b60289e8b12 2023-03-20 00:56:48,418 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/b924b94e8117445e80a133f74ffa7494 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/b924b94e8117445e80a133f74ffa7494 2023-03-20 00:56:48,419 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/382070e20f13462d8dc2280f17ba5500 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/382070e20f13462d8dc2280f17ba5500 2023-03-20 00:56:48,420 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/dab575a7e3d44b4cb5f4a5ffb1541c6c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/dab575a7e3d44b4cb5f4a5ffb1541c6c 2023-03-20 00:56:48,422 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/49f179b5719b4df8b44eebf64d7209f5 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/49f179b5719b4df8b44eebf64d7209f5 2023-03-20 00:56:48,424 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/075c8903933e4cf8808edb1c88d3ded3 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/075c8903933e4cf8808edb1c88d3ded3 2023-03-20 00:56:48,426 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/1cf01d060eb541d2bc782ee67422c6c8 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/1cf01d060eb541d2bc782ee67422c6c8 2023-03-20 00:56:48,427 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/8aa358ef68184298ac3c2d4ebd5a568a to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/8aa358ef68184298ac3c2d4ebd5a568a 2023-03-20 00:56:48,428 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/aef15723df944c42928614a1168dfdb4 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/aef15723df944c42928614a1168dfdb4 2023-03-20 00:56:48,429 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/526e47eccbc341a0a50889458a9f0419 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/526e47eccbc341a0a50889458a9f0419 2023-03-20 00:56:48,431 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/88ba241b942d42d7aa7fdb20121c8364 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/88ba241b942d42d7aa7fdb20121c8364 2023-03-20 00:56:48,433 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/4d9f0c21a9a043029a9c02db2154dccf to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/4d9f0c21a9a043029a9c02db2154dccf 2023-03-20 00:56:48,435 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/e5a7148e11e54c8da91e9a9161ae7ee5 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/e5a7148e11e54c8da91e9a9161ae7ee5 2023-03-20 00:56:48,437 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/aaa3277230034f33b9035b2fe37703f7 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/aaa3277230034f33b9035b2fe37703f7 2023-03-20 00:56:48,467 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/876ce9e9f2c243d7bdac9856ee4ea97c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/876ce9e9f2c243d7bdac9856ee4ea97c 2023-03-20 00:56:48,468 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/5b2c6ba1546f4dcc92447a98b8f4fecb to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/5b2c6ba1546f4dcc92447a98b8f4fecb 2023-03-20 00:56:48,469 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/9d7514ef388145f6b5e4090644049052 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/9d7514ef388145f6b5e4090644049052 2023-03-20 00:56:48,470 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/0f83229aa0364f27b2bfb5fd64e75e3c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/0f83229aa0364f27b2bfb5fd64e75e3c 2023-03-20 00:56:48,471 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/b5ecbdac93db43108a28754ab6030523 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/b5ecbdac93db43108a28754ab6030523 2023-03-20 00:56:48,476 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/ab9d24979c6644e09cceec5bf045d38e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/dc72064e26c24519b62ca55fd64604b2, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/803ade874f1d40c8bfd36d35f0246345, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/75efe15dbbd145028b28980d9ee985e7, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/38bacc889e904b59817c75977d67c995, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/e919b624d6534be39bf82f585fd82f71, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/bd64cb659a56423ab60743556fc573da, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/321bb3c6b2594ba5aa99a04c8abfe62c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/fe22c3bd3d864e08b4a181bbb9ef43d3, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/079f4bb06b2a4e69a32dbd9b19253ab2, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/9f8e5dd90e5e4761aabb69c38d25d4c3, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/2cd5702c446d45d3b326dbb949e00ad5, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/fb671629fddf4bf1be7712cc7a30f7a9, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/36b7864e1fc64c4490444c3dffbb87f6, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/f4f99edd764940d9982418583c3e6505, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/e681ea22ba1a44e0975b4900b234b3cd, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/94024ddf3469441ea57eb02709286f43, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/e82a009353bd4d508a714ce02e689d71, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/30b5eb7d165f484f87b2a71f9c1c0ed6, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/86ade0c058414cf7b8971efc678b9f1c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/97d28e361da441a3abc048fd6913d7a9, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/f8f8020b8ead44f48c6a1f7669ee3562, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/1222c7c1a4b245cc9f725724b6d263bc, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/e30fd4dcecc84f39bd3e717dc3b4f8fe, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/e6357ddb720541478a71a67878fda858, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/91a6b00320ac44ea942c2d5a228c0b8a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/f1e228714c7c49628af618d5c8255739, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/d6f00b87da8f40b49880cdbe89701bfa] to archive 2023-03-20 00:56:48,477 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 00:56:48,479 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/ab9d24979c6644e09cceec5bf045d38e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/ab9d24979c6644e09cceec5bf045d38e 2023-03-20 00:56:48,480 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/dc72064e26c24519b62ca55fd64604b2 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/dc72064e26c24519b62ca55fd64604b2 2023-03-20 00:56:48,481 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/803ade874f1d40c8bfd36d35f0246345 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/803ade874f1d40c8bfd36d35f0246345 2023-03-20 00:56:48,482 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/75efe15dbbd145028b28980d9ee985e7 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/75efe15dbbd145028b28980d9ee985e7 2023-03-20 00:56:48,483 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/38bacc889e904b59817c75977d67c995 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/38bacc889e904b59817c75977d67c995 2023-03-20 00:56:48,484 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/e919b624d6534be39bf82f585fd82f71 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/e919b624d6534be39bf82f585fd82f71 2023-03-20 00:56:48,486 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/bd64cb659a56423ab60743556fc573da to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/bd64cb659a56423ab60743556fc573da 2023-03-20 00:56:48,488 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/321bb3c6b2594ba5aa99a04c8abfe62c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/321bb3c6b2594ba5aa99a04c8abfe62c 2023-03-20 00:56:48,489 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/fe22c3bd3d864e08b4a181bbb9ef43d3 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/fe22c3bd3d864e08b4a181bbb9ef43d3 2023-03-20 00:56:48,491 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/079f4bb06b2a4e69a32dbd9b19253ab2 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/079f4bb06b2a4e69a32dbd9b19253ab2 2023-03-20 00:56:48,495 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/9f8e5dd90e5e4761aabb69c38d25d4c3 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/9f8e5dd90e5e4761aabb69c38d25d4c3 2023-03-20 00:56:48,496 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/2cd5702c446d45d3b326dbb949e00ad5 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/2cd5702c446d45d3b326dbb949e00ad5 2023-03-20 00:56:48,498 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/fb671629fddf4bf1be7712cc7a30f7a9 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/fb671629fddf4bf1be7712cc7a30f7a9 2023-03-20 00:56:48,499 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/36b7864e1fc64c4490444c3dffbb87f6 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/36b7864e1fc64c4490444c3dffbb87f6 2023-03-20 00:56:48,501 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/f4f99edd764940d9982418583c3e6505 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/f4f99edd764940d9982418583c3e6505 2023-03-20 00:56:48,504 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/e681ea22ba1a44e0975b4900b234b3cd to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/e681ea22ba1a44e0975b4900b234b3cd 2023-03-20 00:56:48,506 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/94024ddf3469441ea57eb02709286f43 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/94024ddf3469441ea57eb02709286f43 2023-03-20 00:56:48,507 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/e82a009353bd4d508a714ce02e689d71 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/e82a009353bd4d508a714ce02e689d71 2023-03-20 00:56:48,509 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/30b5eb7d165f484f87b2a71f9c1c0ed6 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/30b5eb7d165f484f87b2a71f9c1c0ed6 2023-03-20 00:56:48,512 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/86ade0c058414cf7b8971efc678b9f1c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/86ade0c058414cf7b8971efc678b9f1c 2023-03-20 00:56:48,513 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/97d28e361da441a3abc048fd6913d7a9 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/97d28e361da441a3abc048fd6913d7a9 2023-03-20 00:56:48,515 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/f8f8020b8ead44f48c6a1f7669ee3562 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/f8f8020b8ead44f48c6a1f7669ee3562 2023-03-20 00:56:48,516 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/1222c7c1a4b245cc9f725724b6d263bc to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/1222c7c1a4b245cc9f725724b6d263bc 2023-03-20 00:56:48,518 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/e30fd4dcecc84f39bd3e717dc3b4f8fe to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/e30fd4dcecc84f39bd3e717dc3b4f8fe 2023-03-20 00:56:48,519 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/e6357ddb720541478a71a67878fda858 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/e6357ddb720541478a71a67878fda858 2023-03-20 00:56:48,521 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/91a6b00320ac44ea942c2d5a228c0b8a to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/91a6b00320ac44ea942c2d5a228c0b8a 2023-03-20 00:56:48,527 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/f1e228714c7c49628af618d5c8255739 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/f1e228714c7c49628af618d5c8255739 2023-03-20 00:56:48,528 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/d6f00b87da8f40b49880cdbe89701bfa to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/d6f00b87da8f40b49880cdbe89701bfa 2023-03-20 00:56:48,532 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/da23df92d8404f5d955401c04ae9374d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/66b123c5a0c54a6b9b429d3b9b90aad6, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/97a5beb404104b80a193fc2dd25b5409, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/2c0ca12c3f134637bc61876ce25198c9, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/96b800f77f414819ad4b52c43de89c94, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/3c7aeea2316042e3b529cf26896e27b8, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/d43d7fe6286a4996a12b0566a6534473, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/cb6eea3068404a5880f3d26992650dc8, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/e1144cb1ff44420095aad397b51b89ae, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/8831a9714ff44778a8429cbd143177b2, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/5d093e40c0984ab4b3efd9163c14fc89, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/2e77589b52ee48729774bd3966e7dd73, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/7204e7b7e2f9425eb8a2e408fbade6ed, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/6417c6f279ad4fbd882903e3f8f66ffb, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/6d8ff18b7f404a3a9f2c8091e9d85950, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/9c5b7fa631314b9e8c9670c2ab1222b4, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/5913a7848d454d1e8fddc74afef7b12e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/11eac65952b34391937104ab314fd4f8, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/261f2b7c2ca941d3a70fdf82ab5b34e3, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/22c6f077ff1046e8bddfe7cd578f38c1, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/37ebecbeb4b94e27b2659047977469af, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/894588811dbd4abcbf60047cca5e6667, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/5643fe50fd094b9f80cb921092b25a25, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/a7520f3c8dab4fb4b10033dac7df9d19, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/44af2bb3ea4549eda3b81a9bebd91e7e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/8fcdad622d8145d18dbcafeacd2912ee, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/00722440e933440ab6dc3b0fb764ddea, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/a1827b019d3a49cca39993d941ed036a] to archive 2023-03-20 00:56:48,534 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 00:56:48,537 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/da23df92d8404f5d955401c04ae9374d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/da23df92d8404f5d955401c04ae9374d 2023-03-20 00:56:48,539 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/66b123c5a0c54a6b9b429d3b9b90aad6 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/66b123c5a0c54a6b9b429d3b9b90aad6 2023-03-20 00:56:48,540 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/97a5beb404104b80a193fc2dd25b5409 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/97a5beb404104b80a193fc2dd25b5409 2023-03-20 00:56:48,542 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/2c0ca12c3f134637bc61876ce25198c9 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/2c0ca12c3f134637bc61876ce25198c9 2023-03-20 00:56:48,543 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/96b800f77f414819ad4b52c43de89c94 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/96b800f77f414819ad4b52c43de89c94 2023-03-20 00:56:48,544 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/3c7aeea2316042e3b529cf26896e27b8 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/3c7aeea2316042e3b529cf26896e27b8 2023-03-20 00:56:48,545 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/d43d7fe6286a4996a12b0566a6534473 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/d43d7fe6286a4996a12b0566a6534473 2023-03-20 00:56:48,548 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/cb6eea3068404a5880f3d26992650dc8 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/cb6eea3068404a5880f3d26992650dc8 2023-03-20 00:56:48,551 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/e1144cb1ff44420095aad397b51b89ae to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/e1144cb1ff44420095aad397b51b89ae 2023-03-20 00:56:48,552 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/8831a9714ff44778a8429cbd143177b2 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/8831a9714ff44778a8429cbd143177b2 2023-03-20 00:56:48,553 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/5d093e40c0984ab4b3efd9163c14fc89 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/5d093e40c0984ab4b3efd9163c14fc89 2023-03-20 00:56:48,554 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/2e77589b52ee48729774bd3966e7dd73 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/2e77589b52ee48729774bd3966e7dd73 2023-03-20 00:56:48,557 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/7204e7b7e2f9425eb8a2e408fbade6ed to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/7204e7b7e2f9425eb8a2e408fbade6ed 2023-03-20 00:56:48,559 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/6417c6f279ad4fbd882903e3f8f66ffb to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/6417c6f279ad4fbd882903e3f8f66ffb 2023-03-20 00:56:48,560 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/6d8ff18b7f404a3a9f2c8091e9d85950 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/6d8ff18b7f404a3a9f2c8091e9d85950 2023-03-20 00:56:48,561 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/9c5b7fa631314b9e8c9670c2ab1222b4 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/9c5b7fa631314b9e8c9670c2ab1222b4 2023-03-20 00:56:48,562 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/5913a7848d454d1e8fddc74afef7b12e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/5913a7848d454d1e8fddc74afef7b12e 2023-03-20 00:56:48,564 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/11eac65952b34391937104ab314fd4f8 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/11eac65952b34391937104ab314fd4f8 2023-03-20 00:56:48,565 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/261f2b7c2ca941d3a70fdf82ab5b34e3 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/261f2b7c2ca941d3a70fdf82ab5b34e3 2023-03-20 00:56:48,567 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/22c6f077ff1046e8bddfe7cd578f38c1 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/22c6f077ff1046e8bddfe7cd578f38c1 2023-03-20 00:56:48,570 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/37ebecbeb4b94e27b2659047977469af to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/37ebecbeb4b94e27b2659047977469af 2023-03-20 00:56:48,572 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/894588811dbd4abcbf60047cca5e6667 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/894588811dbd4abcbf60047cca5e6667 2023-03-20 00:56:48,573 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/5643fe50fd094b9f80cb921092b25a25 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/5643fe50fd094b9f80cb921092b25a25 2023-03-20 00:56:48,575 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/a7520f3c8dab4fb4b10033dac7df9d19 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/a7520f3c8dab4fb4b10033dac7df9d19 2023-03-20 00:56:48,577 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/44af2bb3ea4549eda3b81a9bebd91e7e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/44af2bb3ea4549eda3b81a9bebd91e7e 2023-03-20 00:56:48,579 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/8fcdad622d8145d18dbcafeacd2912ee to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/8fcdad622d8145d18dbcafeacd2912ee 2023-03-20 00:56:48,580 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/00722440e933440ab6dc3b0fb764ddea to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/00722440e933440ab6dc3b0fb764ddea 2023-03-20 00:56:48,582 DEBUG [StoreCloser-TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/a1827b019d3a49cca39993d941ed036a to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/a1827b019d3a49cca39993d941ed036a 2023-03-20 00:56:48,586 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/recovered.edits/2079.seqid, newMaxSeqId=2079, maxSeqId=1 2023-03-20 00:56:48,587 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a. 2023-03-20 00:56:48,587 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 51e645e88856df580ad0de225d47b16a: 2023-03-20 00:56:48,589 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:48,589 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=51e645e88856df580ad0de225d47b16a, regionState=CLOSED 2023-03-20 00:56:48,589 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273808589"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679273808589"}]},"ts":"1679273808589"} 2023-03-20 00:56:48,591 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-03-20 00:56:48,591 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 51e645e88856df580ad0de225d47b16a, server=jenkins-hbase20.apache.org,34553,1679273725199 in 838 msec 2023-03-20 00:56:48,592 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-03-20 00:56:48,592 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=51e645e88856df580ad0de225d47b16a, UNASSIGN in 843 msec 2023-03-20 00:56:48,592 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679273808592"}]},"ts":"1679273808592"} 2023-03-20 00:56:48,593 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-20 00:56:48,594 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-20 00:56:48,595 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 946 msec 2023-03-20 00:56:48,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-20 00:56:48,758 INFO [Listener at localhost.localdomain/43361] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-03-20 00:56:48,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-03-20 00:56:48,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:56:48,761 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:56:48,762 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:56:48,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-20 00:56:48,769 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:48,771 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A, FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B, FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C, FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/recovered.edits] 2023-03-20 00:56:48,773 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/4886b40ac25b4eff9aab97a7b649cf36 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/4886b40ac25b4eff9aab97a7b649cf36 2023-03-20 00:56:48,774 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/a6724d4463824f6aaa9e7aa131e124dc to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/a6724d4463824f6aaa9e7aa131e124dc 2023-03-20 00:56:48,775 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/b1b6323acdbb4dfab73fe9c0a4b9b6e0 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/A/b1b6323acdbb4dfab73fe9c0a4b9b6e0 2023-03-20 00:56:48,776 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/6f4f832204c94b85b42b4fbd38bc7c5f to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/6f4f832204c94b85b42b4fbd38bc7c5f 2023-03-20 00:56:48,777 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/985354926c7f4f16bcf83c40468dd36f to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/985354926c7f4f16bcf83c40468dd36f 2023-03-20 00:56:48,778 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/ff13d52fae21495a9c7a709ea6219a76 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/B/ff13d52fae21495a9c7a709ea6219a76 2023-03-20 00:56:48,780 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/9ce03f4d66a94f529bc6f8a9f8124e64 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/9ce03f4d66a94f529bc6f8a9f8124e64 2023-03-20 00:56:48,780 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/a2dc2ad6ab5f480caa822206b69c3d58 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/a2dc2ad6ab5f480caa822206b69c3d58 2023-03-20 00:56:48,781 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/add7825368334f3eab14395829b96c4b to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/C/add7825368334f3eab14395829b96c4b 2023-03-20 00:56:48,783 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/recovered.edits/2079.seqid to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a/recovered.edits/2079.seqid 2023-03-20 00:56:48,784 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/51e645e88856df580ad0de225d47b16a 2023-03-20 00:56:48,784 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-20 00:56:48,785 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:56:48,786 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-20 00:56:48,787 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-20 00:56:48,788 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:56:48,788 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-20 00:56:48,788 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679273808788"}]},"ts":"9223372036854775807"} 2023-03-20 00:56:48,789 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-20 00:56:48,789 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 51e645e88856df580ad0de225d47b16a, NAME => 'TestAcidGuarantees,,1679273785226.51e645e88856df580ad0de225d47b16a.', STARTKEY => '', ENDKEY => ''}] 2023-03-20 00:56:48,789 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-20 00:56:48,789 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679273808789"}]},"ts":"9223372036854775807"} 2023-03-20 00:56:48,790 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-20 00:56:48,791 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:56:48,791 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 32 msec 2023-03-20 00:56:48,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-20 00:56:48,864 INFO [Listener at localhost.localdomain/43361] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-03-20 00:56:48,881 INFO [Listener at localhost.localdomain/43361] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=232 (was 230) - Thread LEAK? -, OpenFileDescriptor=546 (was 545) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=540 (was 392) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=4258 (was 4855) 2023-03-20 00:56:48,894 INFO [Listener at localhost.localdomain/43361] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=232, OpenFileDescriptor=546, MaxFileDescriptor=60000, SystemLoadAverage=540, ProcessCount=168, AvailableMemoryMB=4258 2023-03-20 00:56:48,896 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-20 00:56:48,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-20 00:56:48,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-20 00:56:48,898 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-20 00:56:48,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-03-20 00:56:48,899 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-20 00:56:48,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-20 00:56:48,900 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:56:48,900 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00 empty. 2023-03-20 00:56:48,901 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:56:48,901 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-20 00:56:49,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-20 00:56:49,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-20 00:56:49,308 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-20 00:56:49,311 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 004290d2dcb18ca400bd2b34f204aa00, NAME => 'TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp 2023-03-20 00:56:49,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-20 00:56:49,719 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 00:56:49,719 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 004290d2dcb18ca400bd2b34f204aa00, disabling compactions & flushes 2023-03-20 00:56:49,719 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:49,719 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:49,719 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. after waiting 0 ms 2023-03-20 00:56:49,719 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:49,719 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:49,719 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:56:49,722 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-20 00:56:49,723 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273809722"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679273809722"}]},"ts":"1679273809722"} 2023-03-20 00:56:49,724 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-20 00:56:49,724 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-20 00:56:49,725 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679273809725"}]},"ts":"1679273809725"} 2023-03-20 00:56:49,726 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-20 00:56:49,728 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=004290d2dcb18ca400bd2b34f204aa00, ASSIGN}] 2023-03-20 00:56:49,729 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=004290d2dcb18ca400bd2b34f204aa00, ASSIGN 2023-03-20 00:56:49,730 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=004290d2dcb18ca400bd2b34f204aa00, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34553,1679273725199; forceNewPlan=false, retain=false 2023-03-20 00:56:49,881 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=004290d2dcb18ca400bd2b34f204aa00, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:49,881 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273809881"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679273809881"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679273809881"}]},"ts":"1679273809881"} 2023-03-20 00:56:49,882 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199}] 2023-03-20 00:56:50,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-20 00:56:50,037 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:50,037 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 004290d2dcb18ca400bd2b34f204aa00, NAME => 'TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.', STARTKEY => '', ENDKEY => ''} 2023-03-20 00:56:50,037 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:56:50,037 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 00:56:50,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:56:50,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:56:50,038 INFO [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:56:50,040 DEBUG [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A 2023-03-20 00:56:50,040 DEBUG [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A 2023-03-20 00:56:50,040 INFO [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 00:56:50,040 INFO [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 004290d2dcb18ca400bd2b34f204aa00 columnFamilyName A 2023-03-20 00:56:50,040 INFO [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] regionserver.HStore(310): Store=004290d2dcb18ca400bd2b34f204aa00/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:56:50,040 INFO [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:56:50,041 DEBUG [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B 2023-03-20 00:56:50,041 DEBUG [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B 2023-03-20 00:56:50,041 INFO [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 00:56:50,042 INFO [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 004290d2dcb18ca400bd2b34f204aa00 columnFamilyName B 2023-03-20 00:56:50,042 INFO [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] regionserver.HStore(310): Store=004290d2dcb18ca400bd2b34f204aa00/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:56:50,042 INFO [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:56:50,043 DEBUG [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C 2023-03-20 00:56:50,043 DEBUG [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C 2023-03-20 00:56:50,043 INFO [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 00:56:50,043 INFO [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 004290d2dcb18ca400bd2b34f204aa00 columnFamilyName C 2023-03-20 00:56:50,044 INFO [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] regionserver.HStore(310): Store=004290d2dcb18ca400bd2b34f204aa00/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:56:50,044 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:50,044 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:56:50,045 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:56:50,046 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-20 00:56:50,047 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:56:50,049 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-20 00:56:50,049 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 004290d2dcb18ca400bd2b34f204aa00; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73716848, jitterRate=0.09846663475036621}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-20 00:56:50,049 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:56:50,050 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00., pid=37, masterSystemTime=1679273810033 2023-03-20 00:56:50,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:50,051 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:50,052 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=004290d2dcb18ca400bd2b34f204aa00, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:50,052 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273810052"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679273810052"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679273810052"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679273810052"}]},"ts":"1679273810052"} 2023-03-20 00:56:50,055 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-03-20 00:56:50,055 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 in 171 msec 2023-03-20 00:56:50,056 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-03-20 00:56:50,056 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=004290d2dcb18ca400bd2b34f204aa00, ASSIGN in 327 msec 2023-03-20 00:56:50,057 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-20 00:56:50,057 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679273810057"}]},"ts":"1679273810057"} 2023-03-20 00:56:50,058 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-20 00:56:50,060 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-20 00:56:50,062 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1640 sec 2023-03-20 00:56:51,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-20 00:56:51,007 INFO [Listener at localhost.localdomain/43361] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-03-20 00:56:51,008 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x1eaa86ba to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:56:51,012 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@74a355de, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:56:51,013 DEBUG [Listener at localhost.localdomain/43361] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:56:51,016 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52550, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:56:51,017 DEBUG [Listener at localhost.localdomain/43361] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-20 00:56:51,018 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50936, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-20 00:56:51,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-20 00:56:51,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-20 00:56:51,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-20 00:56:51,433 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-20 00:56:51,436 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-20 00:56:51,436 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-20 00:56:51,439 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-20 00:56:51,442 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=004290d2dcb18ca400bd2b34f204aa00, REOPEN/MOVE}] 2023-03-20 00:56:51,444 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=004290d2dcb18ca400bd2b34f204aa00, REOPEN/MOVE 2023-03-20 00:56:51,444 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=004290d2dcb18ca400bd2b34f204aa00, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:51,444 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273811444"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679273811444"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679273811444"}]},"ts":"1679273811444"} 2023-03-20 00:56:51,445 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199}] 2023-03-20 00:56:51,597 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:56:51,598 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 004290d2dcb18ca400bd2b34f204aa00, disabling compactions & flushes 2023-03-20 00:56:51,598 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:51,598 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:51,598 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. after waiting 0 ms 2023-03-20 00:56:51,598 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:51,604 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-20 00:56:51,604 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:51,604 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:56:51,604 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 004290d2dcb18ca400bd2b34f204aa00 to self. 2023-03-20 00:56:51,606 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:56:51,606 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=004290d2dcb18ca400bd2b34f204aa00, regionState=CLOSED 2023-03-20 00:56:51,606 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273811606"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679273811606"}]},"ts":"1679273811606"} 2023-03-20 00:56:51,609 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-03-20 00:56:51,609 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 in 162 msec 2023-03-20 00:56:51,609 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=004290d2dcb18ca400bd2b34f204aa00, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,34553,1679273725199; forceNewPlan=false, retain=true 2023-03-20 00:56:51,760 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=004290d2dcb18ca400bd2b34f204aa00, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:51,760 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273811760"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679273811760"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679273811760"}]},"ts":"1679273811760"} 2023-03-20 00:56:51,761 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199}] 2023-03-20 00:56:51,918 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:51,918 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 004290d2dcb18ca400bd2b34f204aa00, NAME => 'TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.', STARTKEY => '', ENDKEY => ''} 2023-03-20 00:56:51,919 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:56:51,919 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 00:56:51,919 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:56:51,919 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:56:51,920 INFO [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:56:51,920 DEBUG [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A 2023-03-20 00:56:51,921 DEBUG [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A 2023-03-20 00:56:51,921 INFO [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 00:56:51,921 INFO [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 004290d2dcb18ca400bd2b34f204aa00 columnFamilyName A 2023-03-20 00:56:51,922 INFO [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] regionserver.HStore(310): Store=004290d2dcb18ca400bd2b34f204aa00/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:56:51,923 INFO [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:56:51,923 DEBUG [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B 2023-03-20 00:56:51,924 DEBUG [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B 2023-03-20 00:56:51,924 INFO [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 00:56:51,924 INFO [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 004290d2dcb18ca400bd2b34f204aa00 columnFamilyName B 2023-03-20 00:56:51,925 INFO [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] regionserver.HStore(310): Store=004290d2dcb18ca400bd2b34f204aa00/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:56:51,925 INFO [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:56:51,926 DEBUG [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C 2023-03-20 00:56:51,926 DEBUG [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C 2023-03-20 00:56:51,926 INFO [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 00:56:51,926 INFO [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 004290d2dcb18ca400bd2b34f204aa00 columnFamilyName C 2023-03-20 00:56:51,927 INFO [StoreOpener-004290d2dcb18ca400bd2b34f204aa00-1] regionserver.HStore(310): Store=004290d2dcb18ca400bd2b34f204aa00/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:56:51,927 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:51,928 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:56:51,929 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:56:51,932 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-20 00:56:51,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:56:51,934 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 004290d2dcb18ca400bd2b34f204aa00; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63181956, jitterRate=-0.05851548910140991}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-20 00:56:51,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:56:51,935 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00., pid=42, masterSystemTime=1679273811914 2023-03-20 00:56:51,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:51,937 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:51,937 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=004290d2dcb18ca400bd2b34f204aa00, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:51,938 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273811937"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679273811937"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679273811937"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679273811937"}]},"ts":"1679273811937"} 2023-03-20 00:56:51,940 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-03-20 00:56:51,940 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 in 178 msec 2023-03-20 00:56:51,941 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-03-20 00:56:51,941 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=004290d2dcb18ca400bd2b34f204aa00, REOPEN/MOVE in 498 msec 2023-03-20 00:56:51,943 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-03-20 00:56:51,943 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 502 msec 2023-03-20 00:56:51,944 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 924 msec 2023-03-20 00:56:51,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-03-20 00:56:51,946 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x1d63a8ce to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:56:51,953 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@57b6628e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:56:51,954 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x5d80e5a8 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:56:51,958 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5da5030e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:56:51,958 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x2ea1a2c1 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:56:51,963 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@74d6bf18, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:56:51,964 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x25185572 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:56:51,976 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@60c5cba, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:56:51,976 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x35f0f398 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:56:51,984 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52249391, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:56:51,984 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x5ba3ce53 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:56:51,992 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ded66ad, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:56:51,993 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x5144fcff to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:56:51,996 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@14d061d2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:56:51,997 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x54da4cc4 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:56:52,009 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@228ff1a4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:56:52,010 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x1280c068 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:56:52,018 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@143a1290, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:56:52,018 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x4af630f2 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:56:52,026 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e613a44, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:56:52,034 DEBUG [hconnection-0x762a93fe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:56:52,035 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52558, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:56:52,036 DEBUG [hconnection-0xb731e97-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:56:52,036 DEBUG [hconnection-0x2ab42cd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:56:52,037 DEBUG [hconnection-0x1f7ff6a7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:56:52,037 DEBUG [hconnection-0xc30c8e8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:56:52,040 DEBUG [hconnection-0x32fbe3a6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:56:52,040 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52574, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:56:52,040 DEBUG [hconnection-0x28d7ae70-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:56:52,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:52,040 DEBUG [hconnection-0x7441bee0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:56:52,040 DEBUG [hconnection-0x2a976bd0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:56:52,040 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52592, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:56:52,040 DEBUG [hconnection-0x312ba21-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:56:52,040 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52590, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:56:52,044 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52604, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:56:52,045 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52622, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:56:52,045 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52654, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:56:52,049 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52632, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:56:52,051 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52616, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:56:52,055 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52646, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:56:52,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:52,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:52,060 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:52,060 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:52,060 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:52,060 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:52,061 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:52,061 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:52,063 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:52,063 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:52,064 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:52,064 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:52,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:52,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:52,075 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:52,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:52,075 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:52,075 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:52,075 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:52,075 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:52,075 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:52,075 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:52,076 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:56:52,076 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 004290d2dcb18ca400bd2b34f204aa00 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-20 00:56:52,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:56:52,087 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. as already flushing 2023-03-20 00:56:52,087 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=A 2023-03-20 00:56:52,087 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:52,088 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=B 2023-03-20 00:56:52,088 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:52,088 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=C 2023-03-20 00:56:52,088 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:52,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:52,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273872124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:52,128 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:52,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273872124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:52,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:52,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:52,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273872125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:52,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:52,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273872126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:52,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273872125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:52,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:52,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273872228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:52,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:52,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273872229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:52,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:52,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273872233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:52,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:52,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:52,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273872233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:52,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273872234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:52,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:52,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273872433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:52,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:52,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273872433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:52,442 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:52,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:52,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273872438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:52,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273872439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:52,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:52,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273872439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:52,507 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303205d85075a63444a7d98add88dcf35e3a1 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303205d85075a63444a7d98add88dcf35e3a1 2023-03-20 00:56:52,743 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:52,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273872737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:52,743 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:52,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273872738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:52,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:52,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273872744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:52,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:52,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273872744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:52,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:52,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273872745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:52,920 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/4cdeb9f033484d0190c426529625eb20 2023-03-20 00:56:52,946 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/6a490f891a7645d3bcb7d272eae79f6e 2023-03-20 00:56:52,980 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/b4c7b3e263f646abb3bd5a6ccb055bef 2023-03-20 00:56:52,986 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/4cdeb9f033484d0190c426529625eb20 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/4cdeb9f033484d0190c426529625eb20 2023-03-20 00:56:52,990 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/4cdeb9f033484d0190c426529625eb20, entries=100, sequenceid=16, filesize=18.4 K 2023-03-20 00:56:52,992 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/6a490f891a7645d3bcb7d272eae79f6e as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/6a490f891a7645d3bcb7d272eae79f6e 2023-03-20 00:56:52,999 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/6a490f891a7645d3bcb7d272eae79f6e, entries=100, sequenceid=16, filesize=9.3 K 2023-03-20 00:56:53,000 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/b4c7b3e263f646abb3bd5a6ccb055bef as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/b4c7b3e263f646abb3bd5a6ccb055bef 2023-03-20 00:56:53,005 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/b4c7b3e263f646abb3bd5a6ccb055bef, entries=100, sequenceid=16, filesize=9.3 K 2023-03-20 00:56:53,006 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 004290d2dcb18ca400bd2b34f204aa00 in 930ms, sequenceid=16, compaction requested=false 2023-03-20 00:56:53,006 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:56:53,006 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:53,006 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:53,006 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:53,006 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,006 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:53,006 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:53,007 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:53,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:53,007 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:53,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:53,008 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:53,008 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:53,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:53,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:53,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:53,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:53,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cbd446a[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:53,009 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:53,009 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:53,010 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:53,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:53,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:53,010 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:53,010 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:53,010 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:53,010 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,010 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:53,011 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:53,011 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:53,011 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:53,011 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:53,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:53,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:53,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:53,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:53,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:53,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:53,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:53,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:53,025 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:53,025 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:53,025 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:53,025 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:53,025 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:53,026 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:53,026 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:53,026 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:53,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:53,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:53,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:53,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:53,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:53,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:53,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:53,028 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:53,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:53,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:53,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:53,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:53,030 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:53,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:53,038 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,038 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:53,038 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:53,038 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,038 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:53,038 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:53,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:53,039 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:53,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:53,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:53,039 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:53,039 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:53,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:53,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:53,039 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:53,039 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:53,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:53,039 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:53,139 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:53,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:53,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:53,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:53,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:53,143 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:53,143 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:53,143 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:53,144 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:53,144 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:53,144 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:53,145 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:53,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:53,145 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:53,145 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,145 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:53,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:53,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:53,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:53,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:53,157 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:53,157 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:53,157 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:53,157 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:53,157 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:53,163 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:53,163 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:53,163 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:53,163 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:56:53,163 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 004290d2dcb18ca400bd2b34f204aa00 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-20 00:56:53,163 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=A 2023-03-20 00:56:53,163 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:53,163 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=B 2023-03-20 00:56:53,163 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:53,164 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=C 2023-03-20 00:56:53,164 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:53,251 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. as already flushing 2023-03-20 00:56:53,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:56:53,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:53,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273873262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:53,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273873263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:53,282 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:53,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273873275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:53,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273873274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273873273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:53,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:53,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273873375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273873375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:53,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:53,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273873384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273873384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:53,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273873387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,584 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:53,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273873582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,584 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:53,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273873582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,597 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:53,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273873593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,597 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:53,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273873593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:53,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273873593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,651 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320d6aa121ef4f1432e8b1e0a8d13663962 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320d6aa121ef4f1432e8b1e0a8d13663962 2023-03-20 00:56:53,673 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/1655bf332ed04b8db4c856c2a4a57622 2023-03-20 00:56:53,888 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:53,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273873885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,892 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:53,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273873887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:53,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273873899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:53,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273873900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:53,905 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:53,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273873901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,123 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-20 00:56:54,133 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/203d0f0795724ebd9bab5ac860413fba 2023-03-20 00:56:54,192 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/0a419803d94840b796bd17d6a04cdcb4 2023-03-20 00:56:54,198 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/1655bf332ed04b8db4c856c2a4a57622 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/1655bf332ed04b8db4c856c2a4a57622 2023-03-20 00:56:54,204 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/1655bf332ed04b8db4c856c2a4a57622, entries=150, sequenceid=36, filesize=25.1 K 2023-03-20 00:56:54,205 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/203d0f0795724ebd9bab5ac860413fba as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/203d0f0795724ebd9bab5ac860413fba 2023-03-20 00:56:54,218 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/203d0f0795724ebd9bab5ac860413fba, entries=150, sequenceid=36, filesize=11.6 K 2023-03-20 00:56:54,218 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/0a419803d94840b796bd17d6a04cdcb4 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/0a419803d94840b796bd17d6a04cdcb4 2023-03-20 00:56:54,228 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/0a419803d94840b796bd17d6a04cdcb4, entries=150, sequenceid=36, filesize=11.6 K 2023-03-20 00:56:54,229 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 004290d2dcb18ca400bd2b34f204aa00 in 1066ms, sequenceid=36, compaction requested=false 2023-03-20 00:56:54,229 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:56:54,229 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:54,229 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:54,229 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:54,229 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,234 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:54,234 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:54,235 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:54,235 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:54,235 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:54,235 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:54,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:54,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3944b3cc[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:54,237 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:54,237 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:54,238 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:54,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:54,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:54,238 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:54,238 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:54,238 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:54,238 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,238 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:54,240 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:54,240 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,240 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:54,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:54,240 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:54,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:54,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:54,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:54,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:54,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:54,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:54,243 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:54,243 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:54,243 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:54,243 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:54,243 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:54,256 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:54,256 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:54,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:54,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:54,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:54,257 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:54,258 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,260 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:54,260 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:54,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:54,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:54,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:54,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:54,260 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:54,260 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:54,260 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:54,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:54,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:54,265 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:54,265 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:54,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:54,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:54,265 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:54,265 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:54,265 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,266 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:54,266 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:54,266 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:54,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:54,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:54,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:54,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:54,361 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:54,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:54,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:54,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:54,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:54,365 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:54,365 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:54,365 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:54,365 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:54,365 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:54,365 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:54,367 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,367 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:54,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:54,367 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,367 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:54,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:54,368 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:54,369 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:54,369 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:54,369 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:54,369 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:54,369 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:54,370 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:54,370 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:54,370 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:54,370 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:56:54,371 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 004290d2dcb18ca400bd2b34f204aa00 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-20 00:56:54,371 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=A 2023-03-20 00:56:54,371 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:54,371 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=B 2023-03-20 00:56:54,371 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:54,371 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=C 2023-03-20 00:56:54,371 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:54,401 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303209a0313ec358643e693b905b05e4a0efd to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303209a0313ec358643e693b905b05e4a0efd 2023-03-20 00:56:54,426 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. as already flushing 2023-03-20 00:56:54,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:56:54,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:54,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273874461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:54,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273874467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:54,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273874469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,478 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:54,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273874470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,479 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:54,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273874471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,577 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:54,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273874572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:54,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273874579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:54,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273874579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:54,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273874579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:54,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273874579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:54,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273874780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,793 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:54,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273874788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,793 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:54,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273874788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,793 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:54,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273874789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,793 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:54,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273874789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:54,811 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=49, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/8ed0cd9329334e48890d0011399069e7 2023-03-20 00:56:55,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:55,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273875089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,100 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:55,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273875094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,100 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:55,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273875095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,105 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:55,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273875098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,105 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:55,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273875099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,260 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/b80a3bea4980411fa9dccbf664ba06fa 2023-03-20 00:56:55,293 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/76e5a42f11f74020b5720d1f554ddc1d 2023-03-20 00:56:55,297 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/8ed0cd9329334e48890d0011399069e7 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/8ed0cd9329334e48890d0011399069e7 2023-03-20 00:56:55,302 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/8ed0cd9329334e48890d0011399069e7, entries=100, sequenceid=49, filesize=18.4 K 2023-03-20 00:56:55,303 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/b80a3bea4980411fa9dccbf664ba06fa as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/b80a3bea4980411fa9dccbf664ba06fa 2023-03-20 00:56:55,307 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/b80a3bea4980411fa9dccbf664ba06fa, entries=100, sequenceid=49, filesize=9.3 K 2023-03-20 00:56:55,308 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/76e5a42f11f74020b5720d1f554ddc1d as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/76e5a42f11f74020b5720d1f554ddc1d 2023-03-20 00:56:55,312 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/76e5a42f11f74020b5720d1f554ddc1d, entries=100, sequenceid=49, filesize=9.3 K 2023-03-20 00:56:55,312 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 004290d2dcb18ca400bd2b34f204aa00 in 941ms, sequenceid=49, compaction requested=true 2023-03-20 00:56:55,313 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:56:55,313 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:55,313 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:55,313 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:55,313 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,313 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:55,313 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:55,314 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,314 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:55,314 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:55,314 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:55,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:55,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5447259f[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:55,316 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:55,316 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:55,317 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:55,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:55,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:55,317 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:55,317 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:55,317 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:55,318 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,318 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:55,318 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:55,318 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:55,318 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:55,319 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:55,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:55,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:55,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:55,322 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:55,322 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:55,322 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:55,322 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:55,322 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:55,333 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:55,333 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:55,333 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:55,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:55,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:55,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:55,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:55,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:55,334 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:55,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:55,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:55,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,335 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:55,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:55,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,337 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:55,337 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:55,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:55,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:55,337 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,337 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:55,337 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:55,337 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,337 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:55,337 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:55,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:55,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:55,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:55,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:55,337 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:55,337 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:55,337 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:55,339 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:55,438 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:55,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:55,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:55,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:55,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:55,442 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:55,442 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:55,442 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:55,442 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:55,442 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:55,442 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:55,445 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,445 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:55,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:55,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:55,445 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,445 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:55,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:55,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:55,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:55,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:55,446 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:55,446 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:55,446 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:55,446 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:55,447 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:55,447 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:55,450 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:55,450 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:55,450 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:55,450 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:56:55,450 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 004290d2dcb18ca400bd2b34f204aa00 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-20 00:56:55,450 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=A 2023-03-20 00:56:55,451 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:55,451 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=B 2023-03-20 00:56:55,451 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:55,452 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=C 2023-03-20 00:56:55,452 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:55,493 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320e049e012efa1434cb2ed1d276d27f7b8 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320e049e012efa1434cb2ed1d276d27f7b8 2023-03-20 00:56:55,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:56:55,601 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. as already flushing 2023-03-20 00:56:55,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:55,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273875614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:55,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273875615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:55,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273875618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:55,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273875618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:55,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273875618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:55,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273875719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:55,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273875719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:55,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273875724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:55,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273875724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:55,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273875725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,904 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=68, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/e6ae9899334f4b70a593429c5cd1eabe 2023-03-20 00:56:55,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:55,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273875924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:55,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273875926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:55,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273875931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:55,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273875931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:55,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:55,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273875931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:56,234 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:56,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273876227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:56,235 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:56,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273876232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:56,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:56,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273876233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:56,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:56,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273876233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:56,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:56,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273876234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:56,316 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/d48830c254424f5683aaa9d04e55cf88 2023-03-20 00:56:56,726 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/b6cfc6960e924c1c8aaa1bc9b5df705c 2023-03-20 00:56:56,729 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/e6ae9899334f4b70a593429c5cd1eabe as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/e6ae9899334f4b70a593429c5cd1eabe 2023-03-20 00:56:56,734 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/e6ae9899334f4b70a593429c5cd1eabe, entries=150, sequenceid=68, filesize=25.1 K 2023-03-20 00:56:56,735 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/d48830c254424f5683aaa9d04e55cf88 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/d48830c254424f5683aaa9d04e55cf88 2023-03-20 00:56:56,741 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:56,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273876736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:56,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:56,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273876740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:56,747 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/d48830c254424f5683aaa9d04e55cf88, entries=150, sequenceid=68, filesize=11.6 K 2023-03-20 00:56:56,748 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/b6cfc6960e924c1c8aaa1bc9b5df705c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/b6cfc6960e924c1c8aaa1bc9b5df705c 2023-03-20 00:56:56,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:56,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273876742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:56,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:56,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273876742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:56,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:56,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273876743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:56,751 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/b6cfc6960e924c1c8aaa1bc9b5df705c, entries=150, sequenceid=68, filesize=11.6 K 2023-03-20 00:56:56,751 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 004290d2dcb18ca400bd2b34f204aa00 in 1301ms, sequenceid=68, compaction requested=true 2023-03-20 00:56:56,751 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:56:56,752 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:56,752 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:56,752 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:56,752 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:56,752 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:56,752 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:56,753 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:56,753 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:56,753 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:56,753 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:56,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:56,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:56,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:56,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@586197c[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:56,755 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:56,755 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:56,757 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:56,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:56,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:56,757 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:56,757 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:56,757 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:56,758 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:56,758 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:56,758 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:56,758 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:56,759 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:56,759 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:56,759 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:56,759 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:56,759 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:56,759 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:56,759 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:56,760 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:56,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:56,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:56,760 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:56,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:56,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:56,761 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:56,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:56,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:56,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:56,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:56,762 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:56,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:56,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:56,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:56,763 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:56,764 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:56,764 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:56,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:56,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:56,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:56,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:56,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:56,765 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:56,765 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:56,765 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:56,765 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:56,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:56,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:56,765 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:56,765 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:56,765 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:56,765 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:56,765 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:56,765 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:56,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:56,865 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:56,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:56,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:56,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:56,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:56,868 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:56,868 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:56,868 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:56,868 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:56,869 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:56,869 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:56,869 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:56,869 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:56,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:56,870 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:56,870 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:56,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:56,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:56,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:56,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:56,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:56,871 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:56,871 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:56,871 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:56,871 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:56,871 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:56,871 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:56,871 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:56,871 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:56,871 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:56:56,871 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 004290d2dcb18ca400bd2b34f204aa00 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-20 00:56:56,871 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=A 2023-03-20 00:56:56,871 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:56,871 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=B 2023-03-20 00:56:56,872 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:56,872 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=C 2023-03-20 00:56:56,872 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:56,883 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032014233b6cb80d40fcb00943f42d6270f6 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032014233b6cb80d40fcb00943f42d6270f6 2023-03-20 00:56:56,890 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=82, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/8c4159f4438046ca8fc3425d8b43c96d 2023-03-20 00:56:57,313 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/123a1b7e532e43dcb672a4c7ad604651 2023-03-20 00:56:57,364 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/85912b4c9d474acbaac1433c16258d63 2023-03-20 00:56:57,374 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/8c4159f4438046ca8fc3425d8b43c96d as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/8c4159f4438046ca8fc3425d8b43c96d 2023-03-20 00:56:57,382 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/8c4159f4438046ca8fc3425d8b43c96d, entries=150, sequenceid=82, filesize=25.1 K 2023-03-20 00:56:57,384 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/123a1b7e532e43dcb672a4c7ad604651 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/123a1b7e532e43dcb672a4c7ad604651 2023-03-20 00:56:57,389 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/123a1b7e532e43dcb672a4c7ad604651, entries=150, sequenceid=82, filesize=11.6 K 2023-03-20 00:56:57,391 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/85912b4c9d474acbaac1433c16258d63 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/85912b4c9d474acbaac1433c16258d63 2023-03-20 00:56:57,397 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/85912b4c9d474acbaac1433c16258d63, entries=150, sequenceid=82, filesize=11.6 K 2023-03-20 00:56:57,398 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 004290d2dcb18ca400bd2b34f204aa00 in 527ms, sequenceid=82, compaction requested=true 2023-03-20 00:56:57,398 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:56:57,399 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:57,399 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:57,399 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:57,399 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,399 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:57,399 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:57,400 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:57,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:57,400 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:57,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:57,401 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,401 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:57,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:57,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:57,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:57,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@435e83b2[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:57,402 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:57,402 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,403 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,403 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:57,403 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:57,403 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:57,403 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,403 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:57,404 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:57,404 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:57,404 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,404 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:57,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:57,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:57,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:57,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:57,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:57,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:57,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:57,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:57,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:57,406 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:57,406 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:57,406 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:57,406 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:57,406 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:57,409 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:57,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:57,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:57,410 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:57,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:57,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:57,411 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:57,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:57,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,413 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:57,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:57,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:57,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:57,413 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,413 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:57,413 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:57,413 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,413 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,413 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:57,413 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,413 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:57,413 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,414 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,414 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:57,414 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:57,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:57,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:57,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:57,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:57,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:57,513 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:57,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:57,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:57,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:57,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:57,541 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:57,541 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:57,541 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:57,541 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:57,542 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,542 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:57,543 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:57,543 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,543 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,544 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:57,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:57,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:57,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:57,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:57,550 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:57,550 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:57,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:57,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:57,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:57,554 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:57,554 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:57,554 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:57,554 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:56:57,554 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:56:57,554 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:57,554 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:57,554 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:57,555 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,555 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:57,555 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:57,558 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,558 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:57,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:57,558 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,559 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:57,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:57,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:57,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:57,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:57,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:57,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f1892c6[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:57,560 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:57,560 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,561 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,561 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:57,561 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:57,561 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:57,562 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,562 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:57,562 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,562 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:57,562 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:57,563 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:57,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:57,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:57,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:57,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:57,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:57,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:57,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:57,565 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:57,565 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:57,565 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:57,565 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:57,565 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:57,567 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,567 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:57,567 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:57,567 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:57,569 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:57,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:57,571 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:57,571 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:57,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:57,571 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,571 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:57,571 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:57,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:57,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:57,571 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:57,571 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:57,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:57,572 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:57,572 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:57,572 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,572 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,572 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,672 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:57,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:57,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:57,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:57,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:57,702 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:57,702 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:57,702 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:57,702 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:57,703 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,703 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:57,703 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:57,703 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:57,704 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,704 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:57,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:57,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:57,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:57,714 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:57,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:57,714 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:57,714 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:57,714 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:57,726 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:57,726 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:57,738 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:57,738 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:56:57,738 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:56:57,738 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:57,738 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:57,738 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:57,738 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,738 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:57,738 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:57,740 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,740 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:57,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:57,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:57,741 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,741 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:57,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:57,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:57,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:57,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:57,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@329e2f67[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:57,742 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:57,742 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,743 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,743 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:57,743 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:57,743 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:57,743 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,743 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:57,744 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,744 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:57,744 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:57,744 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:57,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:57,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:57,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:57,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:57,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:57,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:57,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:57,747 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:57,747 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:57,747 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:57,747 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:57,747 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:57,748 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,748 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:57,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,748 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:57,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:57,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:57,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:57,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,749 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:57,750 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:57,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:57,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,753 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:57,753 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:57,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:57,753 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:57,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:57,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:57,754 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:57,754 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:57,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:57,753 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,754 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:57,754 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:57,754 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,754 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,754 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,754 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:57,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:57,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:57,859 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:57,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:57,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:57,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:57,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:57,882 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:57,882 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:57,882 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:57,882 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:57,882 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,883 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:57,884 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,884 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:57,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:57,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:57,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:57,884 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,885 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:57,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:57,885 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:57,890 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:57,891 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:57,891 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:57,891 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:57,891 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:57,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:56:57,893 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 004290d2dcb18ca400bd2b34f204aa00 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-20 00:56:57,893 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=A 2023-03-20 00:56:57,893 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:57,893 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=B 2023-03-20 00:56:57,893 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:57,894 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=C 2023-03-20 00:56:57,894 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:56:57,902 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:57,902 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:57,902 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:57,902 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:56:57,902 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. as already flushing 2023-03-20 00:56:57,902 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:57,903 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:57,903 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:57,903 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,903 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:57,903 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:57,904 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,904 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:57,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:57,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:57,904 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,905 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:57,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:57,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:57,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:57,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:57,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7668db1[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:57,906 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:57,906 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,907 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,907 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:57,907 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:57,907 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:57,908 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,908 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:57,908 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:57,908 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:57,909 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:57,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:57,909 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:57,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:57,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:57,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:57,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:57,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:57,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:57,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:57,914 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:57,915 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:57,915 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:57,915 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:57,915 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:57,921 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,921 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,921 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:57,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:57,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:57,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:57,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:57,922 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:57,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:57,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:57,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:57,924 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:57,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:57,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,928 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,928 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:57,928 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:57,928 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:57,928 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,928 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:57,928 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:57,928 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:57,928 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:57,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:57,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:57,930 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:57,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:57,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:57,930 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:57,931 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:57,930 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320fb0bd447b2bf4296883906b7f8073db0 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320fb0bd447b2bf4296883906b7f8073db0 2023-03-20 00:56:57,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:57,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:57,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679273817925, max timestamp=1679273817954Num uniques 859; 2023-03-20 00:56:57,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679273817925, max timestamp=1679273817954Num uniques 859; 2023-03-20 00:56:57,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-20 00:56:57,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-20 00:56:57,962 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=94, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/f6aa592362b249a0bebc590d37b4e04d 2023-03-20 00:56:57,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679273817937, max timestamp=1679273817954Num uniques 859; 2023-03-20 00:56:57,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 00:56:58,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:58,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273878002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:58,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273878003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,020 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:58,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273878004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,020 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:58,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273878005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,020 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:58,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273878006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,031 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:58,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:58,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:58,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:58,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:58,045 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:58,045 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:58,045 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:58,045 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:58,045 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:58,045 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:58,046 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:58,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:58,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:58,046 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,047 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,047 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:58,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:58,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:58,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:58,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:58,048 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:58,050 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:58,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:58,050 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:58,050 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:58,050 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:58,052 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:58,052 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:58,052 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:58,052 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:56:58,052 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. as already flushing 2023-03-20 00:56:58,052 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:58,052 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:58,053 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:58,053 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,053 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:58,053 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:58,053 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:58,054 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:58,056 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:58,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:58,056 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:58,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:58,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:58,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:58,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3012ddfc[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:58,057 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:58,057 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:58,066 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:58,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:58,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:58,067 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:58,067 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:58,067 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:58,067 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,067 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:58,071 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,071 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:58,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:58,072 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:58,072 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:58,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:58,075 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:58,075 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:58,075 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:58,075 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:58,075 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:58,083 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:58,083 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:58,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:58,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:58,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:58,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:58,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:58,083 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:58,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:58,083 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:58,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:58,084 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,088 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:58,088 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:58,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:58,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:58,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:58,088 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:58,088 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:58,088 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:58,088 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:58,088 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,088 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:58,088 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:58,088 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:58,088 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:58,089 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:58,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:58,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:58,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273878120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:58,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273878121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:58,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273878121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:58,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273878125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,132 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:58,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273878125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,189 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:58,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:58,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:58,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:58,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:58,193 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:58,194 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:58,194 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:58,194 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:58,194 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:58,194 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:58,195 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:58,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:58,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:58,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:58,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:58,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:58,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:58,196 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:58,196 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:58,196 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:58,196 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:58,196 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:58,196 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,196 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:58,196 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:58,197 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:58,197 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:58,197 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:56:58,197 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. as already flushing 2023-03-20 00:56:58,197 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:58,197 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:58,197 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:58,197 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,197 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:58,197 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:58,198 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,198 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:58,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:58,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:58,198 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:58,198 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:58,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:58,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:58,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:58,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:58,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:58,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b07f79a[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:58,199 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:58,199 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:58,200 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:58,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:58,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:58,200 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:58,200 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:58,200 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:58,200 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,200 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:58,201 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:58,201 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:58,201 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,201 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:58,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:58,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:58,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:58,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:58,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:58,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:58,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:58,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:58,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:58,203 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:58,203 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:58,203 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:58,203 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:58,203 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:58,204 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:58,204 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:58,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:58,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:58,205 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:58,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:58,205 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:58,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:58,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:58,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:58,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:58,207 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:58,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,208 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:58,208 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,208 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:58,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:58,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:58,208 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:58,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:58,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:58,208 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:58,209 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:58,209 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:58,209 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,209 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:58,209 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:58,209 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:58,209 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:58,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:58,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:58,309 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:58,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:58,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:58,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:58,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:58,314 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:58,314 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:58,314 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:58,314 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:58,314 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:58,314 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:58,315 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,315 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:58,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:58,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:58,315 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,315 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:58,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:58,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:58,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:58,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:58,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:58,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:58,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:58,316 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:58,316 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:58,316 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:58,316 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:58,316 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:58,316 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:58,316 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:56:58,316 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. as already flushing 2023-03-20 00:56:58,316 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:58,317 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:56:58,317 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:56:58,317 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,317 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:56:58,317 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:56:58,317 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,317 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:58,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:58,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:58,318 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:58,318 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:56:58,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:58,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:58,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:58,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:58,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:56:58,319 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:56:58,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68a1108a[Count = 0] remaining members to acquire global barrier 2023-03-20 00:56:58,319 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:58,320 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:58,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:58,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:58,320 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:56:58,320 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:56:58,320 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,320 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:56:58,320 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:56:58,321 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,321 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:56:58,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:58,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:58,321 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:58,321 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:56:58,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:58,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:58,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:58,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:58,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:58,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:56:58,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:56:58,323 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:56:58,323 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:56:58,323 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:56:58,323 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:56:58,323 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:56:58,324 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:58,324 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:58,324 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:58,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:58,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:58,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:58,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:56:58,325 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:56:58,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:58,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:58,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:56:58,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:58,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:58,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:56:58,326 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:58,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:56:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:56:58,328 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,328 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:58,328 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:56:58,328 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:58,328 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:58,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:56:58,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:56:58,329 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,329 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:56:58,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:56:58,329 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:56:58,329 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:56:58,329 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:58,329 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:56:58,329 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:58,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:58,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:58,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:56:58,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:56:58,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:58,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273878329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:58,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273878329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:58,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273878330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:58,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273878333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:58,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273878334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,383 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=94 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/61b36a3baa3a4dcdb3cb71477c10722b 2023-03-20 00:56:58,398 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=94 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/80fbcc5f8c5d477193cd9607ba6b13d2 2023-03-20 00:56:58,401 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/f6aa592362b249a0bebc590d37b4e04d as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/f6aa592362b249a0bebc590d37b4e04d 2023-03-20 00:56:58,405 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/f6aa592362b249a0bebc590d37b4e04d, entries=350, sequenceid=94, filesize=52.3 K 2023-03-20 00:56:58,406 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/61b36a3baa3a4dcdb3cb71477c10722b as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/61b36a3baa3a4dcdb3cb71477c10722b 2023-03-20 00:56:58,409 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/61b36a3baa3a4dcdb3cb71477c10722b, entries=150, sequenceid=94, filesize=11.6 K 2023-03-20 00:56:58,409 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/80fbcc5f8c5d477193cd9607ba6b13d2 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/80fbcc5f8c5d477193cd9607ba6b13d2 2023-03-20 00:56:58,413 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/80fbcc5f8c5d477193cd9607ba6b13d2, entries=150, sequenceid=94, filesize=11.6 K 2023-03-20 00:56:58,414 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 004290d2dcb18ca400bd2b34f204aa00 in 521ms, sequenceid=94, compaction requested=true 2023-03-20 00:56:58,414 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:56:58,414 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-20 00:56:58,415 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:58,415 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:58,415 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-20 00:56:58,415 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:58,416 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 168344 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-20 00:56:58,417 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 004290d2dcb18ca400bd2b34f204aa00/A is initiating minor compaction (all files) 2023-03-20 00:56:58,417 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 004290d2dcb18ca400bd2b34f204aa00/A in TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:58,417 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/4cdeb9f033484d0190c426529625eb20, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/1655bf332ed04b8db4c856c2a4a57622, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/8ed0cd9329334e48890d0011399069e7, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/e6ae9899334f4b70a593429c5cd1eabe, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/8c4159f4438046ca8fc3425d8b43c96d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/f6aa592362b249a0bebc590d37b4e04d] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp, totalSize=164.4 K 2023-03-20 00:56:58,417 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 66628 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-20 00:56:58,417 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 4cdeb9f033484d0190c426529625eb20, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1679273812064 2023-03-20 00:56:58,417 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 004290d2dcb18ca400bd2b34f204aa00/B is initiating minor compaction (all files) 2023-03-20 00:56:58,417 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 004290d2dcb18ca400bd2b34f204aa00/B in TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:58,417 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/6a490f891a7645d3bcb7d272eae79f6e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/203d0f0795724ebd9bab5ac860413fba, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/b80a3bea4980411fa9dccbf664ba06fa, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/d48830c254424f5683aaa9d04e55cf88, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/123a1b7e532e43dcb672a4c7ad604651, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/61b36a3baa3a4dcdb3cb71477c10722b] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp, totalSize=65.1 K 2023-03-20 00:56:58,417 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 1655bf332ed04b8db4c856c2a4a57622, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1679273812103 2023-03-20 00:56:58,418 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 6a490f891a7645d3bcb7d272eae79f6e, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1679273812064 2023-03-20 00:56:58,418 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 8ed0cd9329334e48890d0011399069e7, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1679273813273 2023-03-20 00:56:58,418 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 203d0f0795724ebd9bab5ac860413fba, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1679273812103 2023-03-20 00:56:58,419 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting e6ae9899334f4b70a593429c5cd1eabe, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1679273814467 2023-03-20 00:56:58,420 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting b80a3bea4980411fa9dccbf664ba06fa, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1679273813273 2023-03-20 00:56:58,420 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 8c4159f4438046ca8fc3425d8b43c96d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1679273815609 2023-03-20 00:56:58,420 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting d48830c254424f5683aaa9d04e55cf88, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1679273814467 2023-03-20 00:56:58,420 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting f6aa592362b249a0bebc590d37b4e04d, keycount=350, bloomtype=ROW, size=52.3 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1679273817852 2023-03-20 00:56:58,420 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 123a1b7e532e43dcb672a4c7ad604651, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1679273815609 2023-03-20 00:56:58,421 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 61b36a3baa3a4dcdb3cb71477c10722b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1679273817852 2023-03-20 00:56:58,429 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:56:58,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:56:58,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:56:58,433 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 004290d2dcb18ca400bd2b34f204aa00#B#compaction#289 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:58,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:56:58,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:56:58,437 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:56:58,437 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:58,437 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:56:58,437 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:56:58,438 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:58,438 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:58,439 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,439 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:56:58,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:56:58,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:56:58,439 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,439 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:56:58,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:58,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:56:58,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:56:58,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:56:58,441 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:56:58,445 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 004290d2dcb18ca400bd2b34f204aa00#A#compaction#288 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:58,446 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:56:58,446 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:56:58,446 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:56:58,446 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:56:58,446 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:56:58,451 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:56:58,451 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:56:58,451 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:58,451 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:56:58,451 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 004290d2dcb18ca400bd2b34f204aa00 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-03-20 00:56:58,451 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=A 2023-03-20 00:56:58,451 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:56:58,451 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=B 2023-03-20 00:56:58,451 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:56:58,451 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=C 2023-03-20 00:56:58,451 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:56:58,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:56:58,642 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. as already flushing 2023-03-20 00:56:58,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:58,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273878643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:58,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273878645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:58,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273878645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:58,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273878651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:58,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273878652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,759 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:58,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273878753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,759 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:58,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273878753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,759 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:58,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273878758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,765 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:58,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273878760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:58,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273878766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,863 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/fae381505cb94dbfa0c1c871dfdc2365 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/fae381505cb94dbfa0c1c871dfdc2365 2023-03-20 00:56:58,868 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 004290d2dcb18ca400bd2b34f204aa00/B of 004290d2dcb18ca400bd2b34f204aa00 into fae381505cb94dbfa0c1c871dfdc2365(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:58,868 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:56:58,868 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00., storeName=004290d2dcb18ca400bd2b34f204aa00/B, priority=10, startTime=1679273818415; duration=0sec 2023-03-20 00:56:58,868 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:56:58,868 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-20 00:56:58,869 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 66628 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-20 00:56:58,870 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 004290d2dcb18ca400bd2b34f204aa00/C is initiating minor compaction (all files) 2023-03-20 00:56:58,870 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 004290d2dcb18ca400bd2b34f204aa00/C in TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:56:58,870 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/b4c7b3e263f646abb3bd5a6ccb055bef, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/0a419803d94840b796bd17d6a04cdcb4, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/76e5a42f11f74020b5720d1f554ddc1d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/b6cfc6960e924c1c8aaa1bc9b5df705c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/85912b4c9d474acbaac1433c16258d63, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/80fbcc5f8c5d477193cd9607ba6b13d2] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp, totalSize=65.1 K 2023-03-20 00:56:58,870 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting b4c7b3e263f646abb3bd5a6ccb055bef, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1679273812064 2023-03-20 00:56:58,870 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 0a419803d94840b796bd17d6a04cdcb4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1679273812103 2023-03-20 00:56:58,871 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 76e5a42f11f74020b5720d1f554ddc1d, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1679273813273 2023-03-20 00:56:58,871 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting b6cfc6960e924c1c8aaa1bc9b5df705c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1679273814467 2023-03-20 00:56:58,872 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 85912b4c9d474acbaac1433c16258d63, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1679273815609 2023-03-20 00:56:58,872 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 80fbcc5f8c5d477193cd9607ba6b13d2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1679273817852 2023-03-20 00:56:58,879 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320b7e53b3643804374a3dbc1f297265e8a to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320b7e53b3643804374a3dbc1f297265e8a 2023-03-20 00:56:58,881 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 004290d2dcb18ca400bd2b34f204aa00#C#compaction#291 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:56:58,889 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/209296d3e74c410bac399e60283ac1af as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/209296d3e74c410bac399e60283ac1af 2023-03-20 00:56:58,893 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 004290d2dcb18ca400bd2b34f204aa00/C of 004290d2dcb18ca400bd2b34f204aa00 into 209296d3e74c410bac399e60283ac1af(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:56:58,893 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:56:58,893 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00., storeName=004290d2dcb18ca400bd2b34f204aa00/C, priority=10, startTime=1679273818415; duration=0sec 2023-03-20 00:56:58,893 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:58,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:58,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:58,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273878960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273878960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:58,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273878960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:58,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273878967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:58,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:58,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273878969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:59,270 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:59,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273879268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:59,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:59,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273879268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:59,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:59,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273879270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:59,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:59,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273879275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:59,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:59,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273879276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:59,286 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=129, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/d79a1aedb27e469f9271be82ea0f16a7 2023-03-20 00:56:59,677 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/eb96605b9a554661875758c80ff9c654 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/eb96605b9a554661875758c80ff9c654 2023-03-20 00:56:59,681 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 004290d2dcb18ca400bd2b34f204aa00/A of 004290d2dcb18ca400bd2b34f204aa00 into eb96605b9a554661875758c80ff9c654(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-20 00:56:59,681 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:56:59,682 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00., storeName=004290d2dcb18ca400bd2b34f204aa00/A, priority=10, startTime=1679273818414; duration=1sec 2023-03-20 00:56:59,682 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:56:59,707 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/00c6ebd6f92a48608db87453ce5f15f1 2023-03-20 00:56:59,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:59,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273879771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:59,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:59,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273879774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:59,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:59,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273879775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:59,781 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:59,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273879780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:56:59,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:56:59,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273879782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:00,125 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/a81c15cbac2f4279868bb0dea0237396 2023-03-20 00:57:00,129 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/d79a1aedb27e469f9271be82ea0f16a7 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/d79a1aedb27e469f9271be82ea0f16a7 2023-03-20 00:57:00,132 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/d79a1aedb27e469f9271be82ea0f16a7, entries=150, sequenceid=129, filesize=25.1 K 2023-03-20 00:57:00,132 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/00c6ebd6f92a48608db87453ce5f15f1 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/00c6ebd6f92a48608db87453ce5f15f1 2023-03-20 00:57:00,135 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/00c6ebd6f92a48608db87453ce5f15f1, entries=150, sequenceid=129, filesize=11.6 K 2023-03-20 00:57:00,136 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/a81c15cbac2f4279868bb0dea0237396 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/a81c15cbac2f4279868bb0dea0237396 2023-03-20 00:57:00,138 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/a81c15cbac2f4279868bb0dea0237396, entries=150, sequenceid=129, filesize=11.6 K 2023-03-20 00:57:00,139 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 004290d2dcb18ca400bd2b34f204aa00 in 1688ms, sequenceid=129, compaction requested=false 2023-03-20 00:57:00,139 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:57:00,139 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:00,139 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:00,139 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:00,139 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:00,139 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:00,139 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:00,140 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:00,140 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:00,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:00,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:00,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:00,140 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:00,140 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:00,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:00,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:00,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b99032a[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:00,141 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:00,142 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:00,142 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:00,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:00,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:00,142 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:00,142 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:00,142 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:00,142 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:00,142 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:00,143 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:00,143 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:00,143 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:00,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:00,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:00,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:00,143 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:00,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:00,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:00,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:00,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:00,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:00,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:00,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:00,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:00,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:00,145 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:00,145 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:00,145 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:00,145 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:00,145 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:00,146 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:00,146 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:00,146 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:00,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:00,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:00,147 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:00,147 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:00,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:00,149 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:00,149 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:00,149 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:00,149 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:00,149 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:00,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:00,149 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:00,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:00,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:00,149 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:00,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:00,149 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:00,149 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:00,149 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:00,149 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:00,149 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:00,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:00,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:00,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:00,249 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:00,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:00,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:00,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:00,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:00,254 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:00,254 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:00,254 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:00,254 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:00,254 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:00,254 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:00,273 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:00,273 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:00,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:00,274 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:00,274 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:00,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:00,274 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:00,278 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:00,278 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:00,278 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:00,278 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:00,278 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:00,282 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:00,282 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:00,282 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:00,282 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:57:00,282 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 004290d2dcb18ca400bd2b34f204aa00 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-20 00:57:00,283 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=A 2023-03-20 00:57:00,283 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:00,283 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=B 2023-03-20 00:57:00,283 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:00,283 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=C 2023-03-20 00:57:00,283 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:00,321 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032020bc383c40aa4daf8ec2c010ca984794 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032020bc383c40aa4daf8ec2c010ca984794 2023-03-20 00:57:00,733 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=142, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/78a3228a3011403a9208a613a18f2a9d 2023-03-20 00:57:00,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:57:00,784 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. as already flushing 2023-03-20 00:57:00,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679273820798, max timestamp=1679273820805Num uniques 859; 2023-03-20 00:57:00,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679273820798, max timestamp=1679273820805Num uniques 859; 2023-03-20 00:57:00,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-20 00:57:00,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-20 00:57:00,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679273820800, max timestamp=1679273820805Num uniques 859; 2023-03-20 00:57:00,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 00:57:00,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:00,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273880819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:00,830 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:00,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273880820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:00,830 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:00,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273880821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:00,830 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:00,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273880822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:00,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:00,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273880830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:00,933 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:00,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273880930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:00,933 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:00,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273880931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:00,933 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:00,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273880931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:00,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:00,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273880931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:00,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:00,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273880936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:01,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273881134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:01,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273881135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:01,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273881135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:01,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273881135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:01,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273881141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,152 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/372d7b6991a2480e8653e2a4046a2af4 2023-03-20 00:57:01,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:01,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273881441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:01,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:01,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273881441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:01,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273881442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273881441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:01,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273881448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,560 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/7ba7fc30ec3b44e8b3201247d5c9330f 2023-03-20 00:57:01,567 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/78a3228a3011403a9208a613a18f2a9d as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/78a3228a3011403a9208a613a18f2a9d 2023-03-20 00:57:01,578 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/78a3228a3011403a9208a613a18f2a9d, entries=150, sequenceid=142, filesize=25.3 K 2023-03-20 00:57:01,579 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/372d7b6991a2480e8653e2a4046a2af4 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/372d7b6991a2480e8653e2a4046a2af4 2023-03-20 00:57:01,584 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/372d7b6991a2480e8653e2a4046a2af4, entries=150, sequenceid=142, filesize=11.8 K 2023-03-20 00:57:01,585 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/7ba7fc30ec3b44e8b3201247d5c9330f as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/7ba7fc30ec3b44e8b3201247d5c9330f 2023-03-20 00:57:01,589 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/7ba7fc30ec3b44e8b3201247d5c9330f, entries=150, sequenceid=142, filesize=11.8 K 2023-03-20 00:57:01,590 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=136.42 KB/139690 for 004290d2dcb18ca400bd2b34f204aa00 in 1308ms, sequenceid=142, compaction requested=true 2023-03-20 00:57:01,590 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:57:01,590 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:01,590 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:01,590 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:01,590 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,591 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:01,591 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:01,591 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:01,591 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:01,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:01,592 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:01,592 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:01,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:01,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:01,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:01,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:01,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:01,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dc2d898[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:01,595 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:01,595 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:01,595 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:01,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:01,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:01,596 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:01,596 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:01,596 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:01,596 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,596 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:01,596 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,596 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:01,596 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:01,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:01,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:01,597 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:01,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:01,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:01,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:01,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:01,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:01,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:01,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:01,599 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:01,599 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:01,599 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:01,599 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:01,599 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:01,600 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:01,600 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:01,600 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:01,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:01,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:01,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:01,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:01,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:01,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:01,601 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:01,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:01,602 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:01,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:01,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:01,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:01,605 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,605 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:01,605 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:01,605 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:01,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:01,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:01,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:01,605 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:01,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:01,605 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,606 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:01,606 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:01,606 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:01,605 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:01,606 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:01,606 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:01,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:01,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:01,706 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:01,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:01,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:01,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:01,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:01,714 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:01,714 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:01,714 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:01,715 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:01,715 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:01,715 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:01,716 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:01,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:01,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:01,716 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:01,716 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,716 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:01,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:01,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:01,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:01,717 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:01,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:01,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:01,718 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:01,718 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:01,718 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:01,722 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:01,722 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:01,722 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:01,722 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:57:01,722 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 004290d2dcb18ca400bd2b34f204aa00 3/3 column families, dataSize=136.42 KB heapSize=385.38 KB 2023-03-20 00:57:01,722 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=A 2023-03-20 00:57:01,722 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:57:01,722 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=B 2023-03-20 00:57:01,723 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:57:01,723 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=C 2023-03-20 00:57:01,723 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:57:01,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:57:01,953 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. as already flushing 2023-03-20 00:57:01,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:01,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273881958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:01,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273881958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:01,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273881959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:01,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273881960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:01,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:01,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273881961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:02,068 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:02,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273882062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:02,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:02,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273882067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:02,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:02,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273882068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:02,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:02,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273882068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:02,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:02,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273882068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:02,139 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303202017cab7676b48d79c63135cffa10e9c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303202017cab7676b48d79c63135cffa10e9c 2023-03-20 00:57:02,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:02,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273882270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:02,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:02,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273882270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:02,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:02,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273882271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:02,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:02,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273882272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:02,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:02,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273882273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:02,564 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=178, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/df0bfe298b7948a38127bb9b546cae5b 2023-03-20 00:57:02,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:02,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273882576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:02,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:02,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273882576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:02,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:02,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273882576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:02,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:02,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273882576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:02,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:02,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273882578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:02,980 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/537596fdf29f4018bea10e4d1f08c1ea 2023-03-20 00:57:03,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:03,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273883081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:03,088 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:03,088 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:03,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273883081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:03,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273883081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:03,088 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:03,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273883083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:03,088 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:03,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273883085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:03,388 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/3638f581119c4358af88cf5a35868316 2023-03-20 00:57:03,392 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/df0bfe298b7948a38127bb9b546cae5b as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/df0bfe298b7948a38127bb9b546cae5b 2023-03-20 00:57:03,395 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/df0bfe298b7948a38127bb9b546cae5b, entries=150, sequenceid=178, filesize=25.3 K 2023-03-20 00:57:03,396 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/537596fdf29f4018bea10e4d1f08c1ea as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/537596fdf29f4018bea10e4d1f08c1ea 2023-03-20 00:57:03,399 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/537596fdf29f4018bea10e4d1f08c1ea, entries=150, sequenceid=178, filesize=11.8 K 2023-03-20 00:57:03,400 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/3638f581119c4358af88cf5a35868316 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/3638f581119c4358af88cf5a35868316 2023-03-20 00:57:03,406 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/3638f581119c4358af88cf5a35868316, entries=150, sequenceid=178, filesize=11.8 K 2023-03-20 00:57:03,406 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~136.42 KB/139690, heapSize ~385.33 KB/394576, currentSize=46.96 KB/48090 for 004290d2dcb18ca400bd2b34f204aa00 in 1684ms, sequenceid=178, compaction requested=true 2023-03-20 00:57:03,407 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:57:03,407 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:03,407 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:03,407 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:03,407 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:03,407 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:03,407 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:03,408 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:03,408 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:03,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:03,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:03,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:03,408 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:03,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:03,408 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:03,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:03,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:03,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:03,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:03,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:03,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69ed4b9d[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:03,413 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:03,413 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:03,421 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:03,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:03,421 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:03,421 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:03,422 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:03,421 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:03,422 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:03,433 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:03,433 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:03,433 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:03,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:03,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:03,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:03,433 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:03,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:03,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:03,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:03,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:03,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:03,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:03,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:03,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:03,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:03,435 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:03,435 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:03,435 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:03,435 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:03,435 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:03,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:03,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:03,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:03,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:03,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:03,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:03,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:03,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:03,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:03,443 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:03,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:03,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:03,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:03,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:03,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:03,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:03,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:03,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:03,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:03,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:03,458 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:03,459 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:03,459 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:03,459 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:03,459 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:03,459 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:03,459 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:03,459 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:03,459 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:03,460 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:03,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:03,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:03,460 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:03,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:03,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:03,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:03,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:03,460 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:03,460 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:03,560 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:03,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:03,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:03,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:03,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:03,571 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:03,571 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:03,571 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:03,571 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:03,573 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:03,573 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:03,574 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:03,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:03,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:03,574 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:03,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:03,574 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:03,574 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:03,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:03,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:03,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:03,575 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:03,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:03,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:03,578 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:03,578 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:03,578 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:03,598 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:03,598 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:03,598 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:03,598 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:57:03,598 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 004290d2dcb18ca400bd2b34f204aa00 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-20 00:57:03,598 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=A 2023-03-20 00:57:03,598 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:03,598 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=B 2023-03-20 00:57:03,598 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:03,598 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=C 2023-03-20 00:57:03,599 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:03,663 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303208882375595c74ae083569fb0c4667818 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303208882375595c74ae083569fb0c4667818 2023-03-20 00:57:04,080 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=188, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/eb383bf3d20e488494035bcf12f462a2 2023-03-20 00:57:04,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:57:04,092 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. as already flushing 2023-03-20 00:57:04,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679273824110, max timestamp=1679273824120Num uniques 859; 2023-03-20 00:57:04,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679273824110, max timestamp=1679273824120Num uniques 859; 2023-03-20 00:57:04,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 00:57:04,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 00:57:04,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679273824110, max timestamp=1679273824120Num uniques 859; 2023-03-20 00:57:04,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 00:57:04,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:04,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273884143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:04,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273884144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,158 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:04,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273884145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:04,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273884153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:04,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273884154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:04,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273884256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,262 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:04,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273884258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,262 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:04,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273884259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,262 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:04,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273884259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:04,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273884263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:04,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273884463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:04,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273884464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:04,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273884464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:04,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273884465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:04,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273884471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,508 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=188 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/f7bf2dc3a19048388dd4cab3101fafd9 2023-03-20 00:57:04,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:04,779 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:04,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273884769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273884769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,779 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:04,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273884770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:04,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273884771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:04,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273884776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,920 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=188 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/71c4b52b06934cc288208a02013212c8 2023-03-20 00:57:04,924 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/eb383bf3d20e488494035bcf12f462a2 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/eb383bf3d20e488494035bcf12f462a2 2023-03-20 00:57:04,929 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/eb383bf3d20e488494035bcf12f462a2, entries=150, sequenceid=188, filesize=25.3 K 2023-03-20 00:57:04,930 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/f7bf2dc3a19048388dd4cab3101fafd9 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/f7bf2dc3a19048388dd4cab3101fafd9 2023-03-20 00:57:04,934 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/f7bf2dc3a19048388dd4cab3101fafd9, entries=150, sequenceid=188, filesize=11.8 K 2023-03-20 00:57:04,934 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/71c4b52b06934cc288208a02013212c8 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/71c4b52b06934cc288208a02013212c8 2023-03-20 00:57:04,938 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/71c4b52b06934cc288208a02013212c8, entries=150, sequenceid=188, filesize=11.8 K 2023-03-20 00:57:04,939 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 004290d2dcb18ca400bd2b34f204aa00 in 1341ms, sequenceid=188, compaction requested=true 2023-03-20 00:57:04,939 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:57:04,939 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:04,939 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:04,939 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:04,939 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,939 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:04,939 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:04,941 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,941 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:04,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:04,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:04,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:04,942 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:04,942 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:04,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:04,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:04,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:04,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:04,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@928336a[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:04,943 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:04,944 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:04,944 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:04,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:04,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:04,944 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:04,944 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:04,944 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:04,945 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,945 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:04,945 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:04,945 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,945 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:04,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:04,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:04,945 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:04,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:04,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:04,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:04,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:04,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:04,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:04,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:04,948 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:04,948 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:04,948 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:04,948 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:04,948 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:04,950 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:04,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:04,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:04,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:04,950 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:04,950 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:04,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:04,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:04,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:04,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:04,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:04,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:04,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:04,958 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:04,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:04,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:04,959 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:04,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,963 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,963 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:04,963 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:04,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:04,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:04,963 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:04,963 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:04,963 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:04,963 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:04,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:04,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:04,963 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:04,963 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:04,963 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:04,963 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:04,963 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:04,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:04,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:05,063 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:05,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:05,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:05,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:05,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:05,066 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:05,066 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:05,066 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:05,066 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:05,067 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,067 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:05,067 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:05,067 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:05,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:05,068 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,068 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:05,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:05,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:05,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:05,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:05,070 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:05,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:05,070 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:05,070 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:05,070 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:05,070 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:05,070 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:05,070 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:05,070 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:57:05,070 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 004290d2dcb18ca400bd2b34f204aa00 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-20 00:57:05,070 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=A 2023-03-20 00:57:05,071 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:57:05,071 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=B 2023-03-20 00:57:05,071 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:57:05,071 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=C 2023-03-20 00:57:05,071 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:57:05,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:57:05,283 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. as already flushing 2023-03-20 00:57:05,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:05,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273885291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,300 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:05,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273885291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,300 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:05,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273885292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,300 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:05,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273885295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,302 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:05,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273885295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,399 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:05,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273885396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:05,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273885401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:05,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273885401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:05,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273885401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:05,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273885403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,482 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320477ca1cb6f28439b9e5cab1ba4e9899b to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320477ca1cb6f28439b9e5cab1ba4e9899b 2023-03-20 00:57:05,488 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=224, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/3340dd188c72466cb83dca8a34c47466 2023-03-20 00:57:05,504 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/0a95a3d0c87c4f3e9b448b5c979f9b9d 2023-03-20 00:57:05,513 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/f13dc0dd4fde4820b97f61a5a6785b74 2023-03-20 00:57:05,516 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/3340dd188c72466cb83dca8a34c47466 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/3340dd188c72466cb83dca8a34c47466 2023-03-20 00:57:05,519 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/3340dd188c72466cb83dca8a34c47466, entries=150, sequenceid=224, filesize=25.3 K 2023-03-20 00:57:05,520 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/0a95a3d0c87c4f3e9b448b5c979f9b9d as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/0a95a3d0c87c4f3e9b448b5c979f9b9d 2023-03-20 00:57:05,523 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/0a95a3d0c87c4f3e9b448b5c979f9b9d, entries=150, sequenceid=224, filesize=11.8 K 2023-03-20 00:57:05,523 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/f13dc0dd4fde4820b97f61a5a6785b74 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/f13dc0dd4fde4820b97f61a5a6785b74 2023-03-20 00:57:05,527 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/f13dc0dd4fde4820b97f61a5a6785b74, entries=150, sequenceid=224, filesize=11.8 K 2023-03-20 00:57:05,527 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=53.67 KB/54960 for 004290d2dcb18ca400bd2b34f204aa00 in 457ms, sequenceid=224, compaction requested=true 2023-03-20 00:57:05,527 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:57:05,528 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:05,528 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:05,528 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:05,528 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,528 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:05,528 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:05,529 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,529 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,529 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:05,530 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:05,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:05,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:05,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:05,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:05,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:05,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:05,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1abbe136[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:05,533 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:05,533 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,533 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,533 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:05,533 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:05,533 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:05,534 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,534 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:05,534 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,534 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:05,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:05,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:05,534 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:05,534 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:05,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:05,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:05,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:05,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:05,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:05,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:05,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:05,537 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:05,537 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:05,537 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:05,537 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:05,537 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:05,538 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,538 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,538 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:05,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:05,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:05,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:05,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:05,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:05,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,544 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:05,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:05,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:05,545 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:05,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:05,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,554 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,555 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:05,555 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:05,555 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:05,555 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:05,555 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:05,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:05,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:05,555 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,555 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,555 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:05,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:05,555 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:05,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:05,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:05,556 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:05,556 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:05,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:57:05,603 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 004290d2dcb18ca400bd2b34f204aa00 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-20 00:57:05,603 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=A 2023-03-20 00:57:05,603 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:05,603 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=B 2023-03-20 00:57:05,604 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:05,604 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=C 2023-03-20 00:57:05,604 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:05,636 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:05,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273885627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,636 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:05,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52604 deadline: 1679273885628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:05,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:05,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273885633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273885633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,646 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:05,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273885635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,656 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:05,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:05,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:05,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:05,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:05,658 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:05,658 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:05,659 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:05,659 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:05,659 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,659 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:05,660 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,660 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:05,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:05,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:05,660 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,660 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:05,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:05,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:05,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:05,662 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:05,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:05,662 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:05,662 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:05,662 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:05,662 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:05,662 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:05,662 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:05,662 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:05,662 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:57:05,662 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. as already flushing 2023-03-20 00:57:05,662 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:05,662 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:05,662 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:05,663 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,663 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:05,663 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:05,663 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,663 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:05,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:05,664 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,664 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:05,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:05,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:05,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:05,668 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:05,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71d6fce1[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:05,668 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,669 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,669 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:05,669 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,669 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:05,669 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:05,669 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:05,670 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:05,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:05,670 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:05,670 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:05,670 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:05,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:05,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:05,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:05,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:05,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:05,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:05,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:05,672 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:05,672 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:05,672 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:05,672 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:05,672 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:05,673 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:05,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:05,673 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,673 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:05,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:05,674 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:05,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:05,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:05,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:05,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:05,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,675 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:05,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:05,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,683 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,683 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:05,683 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:05,683 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:05,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:05,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:05,683 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:05,683 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,683 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,683 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,683 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,683 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:05,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:05,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:05,683 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:05,683 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:05,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:05,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:05,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:05,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52604 deadline: 1679273885737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:05,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273885737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:05,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273885746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:05,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52558 deadline: 1679273885747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:05,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273885747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,784 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:05,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:05,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:05,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:05,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:05,787 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:05,787 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:05,787 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:05,787 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:05,787 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,787 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:05,788 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,788 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:05,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:05,788 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,788 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:05,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:05,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:05,789 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:05,789 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:05,789 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:05,789 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:05,789 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:05,789 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:05,789 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:05,789 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:05,789 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:05,789 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:57:05,790 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. as already flushing 2023-03-20 00:57:05,790 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:05,790 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:05,790 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:05,790 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,790 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:05,790 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:05,790 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,790 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:05,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:05,791 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,791 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:05,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:05,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:05,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:05,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:05,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:05,793 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:05,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72e4e0e5[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:05,793 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,795 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,795 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:05,796 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:05,796 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:05,796 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:05,796 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:05,796 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:05,799 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,799 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:05,799 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:05,799 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:05,799 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:05,799 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:05,799 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:05,799 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:05,808 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,808 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,808 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:05,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:05,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:05,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:05,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:05,809 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:05,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:05,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:05,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:05,812 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:05,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:05,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,814 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:05,814 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:05,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:05,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:05,814 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,814 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:05,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:05,814 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:05,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:05,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:05,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:05,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:05,814 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:05,814 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,814 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,814 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:05,815 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,815 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:05,815 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,915 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:05,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:05,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:05,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:05,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:05,930 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:05,930 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:05,930 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:05,930 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:05,930 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,930 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:05,931 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:05,931 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:05,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:05,931 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,931 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:05,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:05,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:05,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:05,932 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:05,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:05,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:05,934 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:05,934 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:05,934 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:05,938 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:05,938 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:05,938 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:05,938 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:57:05,938 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. as already flushing 2023-03-20 00:57:05,938 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:05,939 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:05,939 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:05,939 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,939 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:05,939 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:05,941 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,941 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:05,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:05,941 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,941 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:05,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:05,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:05,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:05,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:05,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:05,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a1f2eb0[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:05,943 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:05,943 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,943 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,943 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:05,943 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:05,943 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,943 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:05,943 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:05,944 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:05,944 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:05,944 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,944 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:05,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:05,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:05,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:05,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:05,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:05,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,945 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:05,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52604 deadline: 1679273885942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:05,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52622 deadline: 1679273885942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:05,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:05,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:05,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:05,947 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:05,947 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:05,948 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:05,948 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:05,948 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:05,948 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,948 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,948 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:05,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:05,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:05,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:05,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:05,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:05,949 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:05,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:05,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:05,950 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:05,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:05,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,951 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,951 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:05,951 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:05,951 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,951 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:05,951 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:05,951 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:05,951 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:05,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:05,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:05,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:05,951 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:05,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:05,952 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:05,952 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:05,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:05,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:05,952 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:05,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:05,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52558 deadline: 1679273885952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:05,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52574 deadline: 1679273885953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:05,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:05,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52616 deadline: 1679273885953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,019 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320d3c01fed1af94425a958d077a0cfe6cc to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320d3c01fed1af94425a958d077a0cfe6cc 2023-03-20 00:57:06,052 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:06,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:06,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:06,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:06,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:06,057 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:06,057 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:06,057 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:06,057 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:06,058 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,058 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,058 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:06,058 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:06,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:06,059 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,059 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:06,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:06,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:06,063 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:06,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:06,064 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:06,064 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:06,064 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:06,065 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:06,065 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:06,065 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:06,065 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:57:06,065 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. as already flushing 2023-03-20 00:57:06,065 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:06,065 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:06,065 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:06,065 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,065 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:06,065 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:06,066 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,067 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,067 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:06,067 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:06,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:06,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:06,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:06,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:06,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:06,069 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:06,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f68da94[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:06,069 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,069 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,069 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:06,069 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:06,069 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:06,069 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,070 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:06,070 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,070 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:06,070 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:06,070 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:06,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:06,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:06,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:06,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:06,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:06,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:06,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:06,073 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:06,073 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:06,073 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:06,073 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:06,073 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:06,074 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,074 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,074 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:06,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:06,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:06,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,075 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:06,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:06,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:06,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:06,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,076 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:06,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,077 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:06,077 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:06,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:06,077 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:06,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:06,077 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,077 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:06,077 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:06,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:06,077 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:06,077 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,077 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:06,077 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,078 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:06,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:06,078 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,178 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:06,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:06,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:06,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:06,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:06,180 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:06,181 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:06,181 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:06,181 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:06,181 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,181 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,181 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,181 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:06,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:06,182 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,182 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,182 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:06,182 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:06,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:06,183 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:06,183 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:06,183 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:06,183 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:06,183 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:06,183 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:06,183 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:57:06,183 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. as already flushing 2023-03-20 00:57:06,183 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:06,183 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:06,183 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:06,183 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,183 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:06,183 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:06,184 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,184 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:06,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:06,184 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,184 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:06,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:06,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:06,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:06,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:06,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@399f106c[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:06,186 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:06,186 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,187 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,187 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:06,187 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:06,187 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,187 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:06,187 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:06,189 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,189 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:06,189 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:06,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:06,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:06,189 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:06,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:06,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:06,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:06,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:06,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:06,191 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:06,191 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:06,191 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:06,191 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:06,191 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:06,192 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,192 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:06,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,192 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:06,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:06,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:06,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:06,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:06,192 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:06,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,194 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:06,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,195 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:06,195 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:06,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:06,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:06,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:06,195 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,195 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:06,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:06,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:06,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:06,195 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,195 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:06,196 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,196 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:06,196 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,196 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:06,196 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,196 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:06,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52622 deadline: 1679273886247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:06,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52604 deadline: 1679273886248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,266 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:06,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52558 deadline: 1679273886261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,266 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:06,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52574 deadline: 1679273886261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,270 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:06,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52616 deadline: 1679273886263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,296 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:06,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:06,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:06,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:06,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:06,299 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:06,299 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:06,299 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:06,299 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:06,299 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,299 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,300 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:06,300 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:06,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:06,300 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,300 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:06,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:06,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,301 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:06,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:06,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:06,301 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:06,301 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:06,301 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:06,301 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:06,301 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:06,302 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:06,302 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:57:06,302 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. as already flushing 2023-03-20 00:57:06,302 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:06,302 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:06,302 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:06,302 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,302 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:06,302 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:06,303 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,303 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:06,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:06,303 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,303 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:06,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:06,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:06,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:06,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:06,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ca92c72[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:06,304 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:06,304 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,305 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,305 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:06,305 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:06,305 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:06,305 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,305 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:06,305 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,305 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:06,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:06,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:06,306 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:06,306 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:06,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:06,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:06,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:06,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:06,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:06,307 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:06,307 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:06,307 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:06,307 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:06,307 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:06,308 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,308 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:06,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:06,308 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:06,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:06,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:06,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:06,309 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:06,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,310 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:06,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,312 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:06,312 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:06,312 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,312 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:06,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:06,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:06,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:06,312 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:06,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:06,312 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:06,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:06,312 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,312 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:06,312 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:06,312 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,312 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,312 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,412 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:06,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:06,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:06,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:06,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:06,415 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:06,415 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:06,415 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:06,415 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:06,415 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,415 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,416 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:06,416 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:06,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:06,416 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,416 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:06,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:06,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,417 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:06,418 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:06,418 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:06,418 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:06,418 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:06,418 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:06,418 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:06,418 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:06,418 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:06,418 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:57:06,418 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. as already flushing 2023-03-20 00:57:06,418 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:06,418 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:06,418 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:06,418 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,418 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:06,418 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:06,419 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,419 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:06,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:06,419 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,419 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:06,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:06,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:06,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:06,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:06,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f80e5f9[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:06,421 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:06,421 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,421 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,421 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:06,421 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:06,421 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,421 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:06,421 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:06,422 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,422 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:06,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:06,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:06,422 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:06,422 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:06,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:06,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:06,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:06,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:06,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:06,424 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:06,424 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:06,424 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:06,424 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:06,424 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:06,436 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=236, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/4fa2998a65084e11b334283720c2d13e 2023-03-20 00:57:06,438 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:06,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:06,438 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,438 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:06,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:06,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:06,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:06,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,439 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:06,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,440 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:06,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,442 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,442 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:06,442 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:06,442 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,442 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,442 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,442 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:06,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:06,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:06,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:06,442 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,443 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,443 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:06,443 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:06,443 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:06,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:06,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:06,543 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:06,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:06,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:06,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:06,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:06,547 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:06,547 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:06,547 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:06,547 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:06,547 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,547 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,548 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:06,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:06,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:06,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,549 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:06,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:06,550 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,550 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:06,550 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:06,551 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:06,551 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:06,551 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:06,551 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:06,551 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:06,551 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:06,551 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:06,551 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:57:06,551 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. as already flushing 2023-03-20 00:57:06,551 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:06,551 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:06,551 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:06,551 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,551 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:06,551 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:06,552 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:06,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:06,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:06,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:06,553 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,554 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,554 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:06,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:06,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:06,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@160526e7[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:06,556 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:06,556 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,557 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,557 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:06,557 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:06,557 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:06,557 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,557 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:06,558 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:06,558 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,558 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:06,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:06,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:06,558 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:06,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:06,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:06,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:06,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:06,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:06,561 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:06,561 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:06,561 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:06,561 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:06,561 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:06,562 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:06,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:06,562 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,562 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:06,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:06,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:06,563 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:06,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:06,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,564 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:06,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,566 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:06,566 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:06,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:06,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:06,566 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,567 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,567 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:06,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:06,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:06,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:06,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:06,567 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,567 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,567 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:06,567 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,567 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:06,567 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,567 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,667 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:06,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:06,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:06,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:06,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:06,670 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:06,670 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:06,670 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:06,670 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:06,670 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,670 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,671 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,671 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:06,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:06,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:06,671 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,671 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:06,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:06,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:06,672 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:06,672 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:06,672 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:06,672 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:06,672 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:06,672 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:06,672 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:06,672 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:06,672 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:57:06,672 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. as already flushing 2023-03-20 00:57:06,672 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:06,672 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:06,673 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:06,673 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,673 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:06,673 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:06,673 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,673 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:06,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:06,674 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,674 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:06,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:06,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:06,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:06,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:06,680 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:06,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5758483b[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:06,680 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,696 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,696 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:06,696 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:06,696 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:06,698 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,698 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:06,699 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:06,699 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:06,699 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,699 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:06,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:06,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:06,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:06,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:06,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:06,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:06,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:06,701 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:06,701 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:06,701 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:06,701 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:06,701 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:06,702 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,702 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,702 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:06,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:06,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:06,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,702 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:06,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:06,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:06,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:06,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,703 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:06,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,706 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,706 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,706 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,706 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,706 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,707 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,707 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,707 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:06,707 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:06,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:06,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:06,707 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:06,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:06,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:06,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:06,707 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:06,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:06,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:06,707 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:06,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:06,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52622 deadline: 1679273886754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:06,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52604 deadline: 1679273886754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:06,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52574 deadline: 1679273886767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:06,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52558 deadline: 1679273886768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,780 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:06,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52616 deadline: 1679273886774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,808 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:06,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:06,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:06,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:06,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:06,810 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:06,810 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:06,811 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:06,811 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:06,811 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,811 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,811 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:06,811 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:06,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:06,812 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,812 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:06,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:06,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,812 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:06,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:06,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:06,813 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:06,813 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:06,813 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:06,813 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:06,813 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:06,813 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:06,813 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:57:06,813 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. as already flushing 2023-03-20 00:57:06,813 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:06,813 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:06,813 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:06,813 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,813 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:06,813 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:06,814 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,814 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:06,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:06,814 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,814 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:06,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:06,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:06,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:06,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:06,815 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:06,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72b7341e[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:06,816 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,816 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,816 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:06,816 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,816 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:06,816 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:06,816 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:06,817 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:06,817 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,817 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:06,817 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:06,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:06,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:06,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:06,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:06,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:06,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:06,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:06,820 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:06,820 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:06,820 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:06,820 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:06,820 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:06,821 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,821 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,821 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:06,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:06,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:06,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:06,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:06,822 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:06,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:06,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,823 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:06,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:06,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,824 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:06,824 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:06,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:06,824 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:06,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:06,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:06,824 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:06,824 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:06,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:06,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:06,825 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:06,825 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,825 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:06,825 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,825 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,825 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:06,825 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,851 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=236 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/793737a205dc44e182d9b5fad59a64d2 2023-03-20 00:57:06,860 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=236 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/ea101ef110d449b19a94234487dbf4ea 2023-03-20 00:57:06,866 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/4fa2998a65084e11b334283720c2d13e as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/4fa2998a65084e11b334283720c2d13e 2023-03-20 00:57:06,870 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/4fa2998a65084e11b334283720c2d13e, entries=200, sequenceid=236, filesize=32.1 K 2023-03-20 00:57:06,871 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/793737a205dc44e182d9b5fad59a64d2 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/793737a205dc44e182d9b5fad59a64d2 2023-03-20 00:57:06,881 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/793737a205dc44e182d9b5fad59a64d2, entries=150, sequenceid=236, filesize=11.8 K 2023-03-20 00:57:06,882 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/ea101ef110d449b19a94234487dbf4ea as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/ea101ef110d449b19a94234487dbf4ea 2023-03-20 00:57:06,887 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/ea101ef110d449b19a94234487dbf4ea, entries=150, sequenceid=236, filesize=11.8 K 2023-03-20 00:57:06,888 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 004290d2dcb18ca400bd2b34f204aa00 in 1285ms, sequenceid=236, compaction requested=true 2023-03-20 00:57:06,888 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:57:06,888 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:06,888 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-20 00:57:06,888 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:06,888 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-20 00:57:06,888 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:06,890 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 188168 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-20 00:57:06,890 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 84158 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-20 00:57:06,890 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 004290d2dcb18ca400bd2b34f204aa00/A is initiating minor compaction (all files) 2023-03-20 00:57:06,890 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 004290d2dcb18ca400bd2b34f204aa00/B is initiating minor compaction (all files) 2023-03-20 00:57:06,890 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 004290d2dcb18ca400bd2b34f204aa00/A in TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:06,890 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 004290d2dcb18ca400bd2b34f204aa00/B in TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:06,890 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/fae381505cb94dbfa0c1c871dfdc2365, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/00c6ebd6f92a48608db87453ce5f15f1, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/372d7b6991a2480e8653e2a4046a2af4, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/537596fdf29f4018bea10e4d1f08c1ea, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/f7bf2dc3a19048388dd4cab3101fafd9, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/0a95a3d0c87c4f3e9b448b5c979f9b9d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/793737a205dc44e182d9b5fad59a64d2] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp, totalSize=82.2 K 2023-03-20 00:57:06,890 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/eb96605b9a554661875758c80ff9c654, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/d79a1aedb27e469f9271be82ea0f16a7, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/78a3228a3011403a9208a613a18f2a9d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/df0bfe298b7948a38127bb9b546cae5b, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/eb383bf3d20e488494035bcf12f462a2, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/3340dd188c72466cb83dca8a34c47466, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/4fa2998a65084e11b334283720c2d13e] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp, totalSize=183.8 K 2023-03-20 00:57:06,890 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting fae381505cb94dbfa0c1c871dfdc2365, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1679273817852 2023-03-20 00:57:06,890 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting eb96605b9a554661875758c80ff9c654, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1679273817852 2023-03-20 00:57:06,891 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 00c6ebd6f92a48608db87453ce5f15f1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1679273817977 2023-03-20 00:57:06,891 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting d79a1aedb27e469f9271be82ea0f16a7, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1679273817977 2023-03-20 00:57:06,891 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 372d7b6991a2480e8653e2a4046a2af4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1679273818642 2023-03-20 00:57:06,891 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 78a3228a3011403a9208a613a18f2a9d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1679273818642 2023-03-20 00:57:06,891 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 537596fdf29f4018bea10e4d1f08c1ea, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1679273820819 2023-03-20 00:57:06,891 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting df0bfe298b7948a38127bb9b546cae5b, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1679273820819 2023-03-20 00:57:06,891 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting f7bf2dc3a19048388dd4cab3101fafd9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=188, earliestPutTs=1679273821957 2023-03-20 00:57:06,892 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting eb383bf3d20e488494035bcf12f462a2, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=188, earliestPutTs=1679273821957 2023-03-20 00:57:06,892 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 0a95a3d0c87c4f3e9b448b5c979f9b9d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1679273824142 2023-03-20 00:57:06,892 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 3340dd188c72466cb83dca8a34c47466, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1679273824142 2023-03-20 00:57:06,892 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 793737a205dc44e182d9b5fad59a64d2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1679273825291 2023-03-20 00:57:06,892 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 4fa2998a65084e11b334283720c2d13e, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1679273825291 2023-03-20 00:57:06,927 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 004290d2dcb18ca400bd2b34f204aa00#B#compaction#310 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:06,928 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:06,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:06,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:06,938 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 004290d2dcb18ca400bd2b34f204aa00#A#compaction#309 average throughput is 0.56 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:06,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:06,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:06,938 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:06,938 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:06,938 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:06,938 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:06,940 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,940 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,940 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:06,940 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:06,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:06,941 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:06,941 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:06,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:06,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:06,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:06,942 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:06,943 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:06,943 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:06,943 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:06,943 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:06,943 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:06,943 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:06,943 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:06,944 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:06,944 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:57:06,944 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 004290d2dcb18ca400bd2b34f204aa00 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-20 00:57:06,944 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=A 2023-03-20 00:57:06,944 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:06,944 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=B 2023-03-20 00:57:06,944 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:06,944 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=C 2023-03-20 00:57:06,944 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:06,952 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/9ca1eb4b32f24abbb42ae7564f8e0c1d as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/9ca1eb4b32f24abbb42ae7564f8e0c1d 2023-03-20 00:57:06,960 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 004290d2dcb18ca400bd2b34f204aa00/B of 004290d2dcb18ca400bd2b34f204aa00 into 9ca1eb4b32f24abbb42ae7564f8e0c1d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:06,960 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:57:06,960 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00., storeName=004290d2dcb18ca400bd2b34f204aa00/B, priority=9, startTime=1679273826888; duration=0sec 2023-03-20 00:57:06,960 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:06,960 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-20 00:57:06,964 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 84158 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-20 00:57:06,964 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 004290d2dcb18ca400bd2b34f204aa00/C is initiating minor compaction (all files) 2023-03-20 00:57:06,965 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 004290d2dcb18ca400bd2b34f204aa00/C in TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:06,965 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/209296d3e74c410bac399e60283ac1af, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/a81c15cbac2f4279868bb0dea0237396, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/7ba7fc30ec3b44e8b3201247d5c9330f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/3638f581119c4358af88cf5a35868316, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/71c4b52b06934cc288208a02013212c8, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/f13dc0dd4fde4820b97f61a5a6785b74, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/ea101ef110d449b19a94234487dbf4ea] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp, totalSize=82.2 K 2023-03-20 00:57:06,966 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 209296d3e74c410bac399e60283ac1af, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1679273817852 2023-03-20 00:57:06,967 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting a81c15cbac2f4279868bb0dea0237396, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1679273817977 2023-03-20 00:57:06,975 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 7ba7fc30ec3b44e8b3201247d5c9330f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1679273818642 2023-03-20 00:57:06,975 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 3638f581119c4358af88cf5a35868316, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1679273820819 2023-03-20 00:57:06,976 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 71c4b52b06934cc288208a02013212c8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=188, earliestPutTs=1679273821957 2023-03-20 00:57:06,979 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting f13dc0dd4fde4820b97f61a5a6785b74, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1679273824142 2023-03-20 00:57:06,979 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting ea101ef110d449b19a94234487dbf4ea, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1679273825291 2023-03-20 00:57:06,997 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320c0bdcb94dfb546c09caade61406137f4 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320c0bdcb94dfb546c09caade61406137f4 2023-03-20 00:57:07,013 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/c4dd0adb5f6a429fb0bb2fb9bb283478 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/c4dd0adb5f6a429fb0bb2fb9bb283478 2023-03-20 00:57:07,031 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 004290d2dcb18ca400bd2b34f204aa00#C#compaction#312 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:07,033 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 004290d2dcb18ca400bd2b34f204aa00/A of 004290d2dcb18ca400bd2b34f204aa00 into c4dd0adb5f6a429fb0bb2fb9bb283478(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:07,033 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:57:07,033 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00., storeName=004290d2dcb18ca400bd2b34f204aa00/A, priority=9, startTime=1679273826888; duration=0sec 2023-03-20 00:57:07,034 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:07,058 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=256, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/3892412c373b479fb9b1d613fc34de29 2023-03-20 00:57:07,097 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/cdc4669909e448e78721efe4bf31e3ed as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/cdc4669909e448e78721efe4bf31e3ed 2023-03-20 00:57:07,109 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 004290d2dcb18ca400bd2b34f204aa00/C of 004290d2dcb18ca400bd2b34f204aa00 into cdc4669909e448e78721efe4bf31e3ed(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:07,109 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:57:07,109 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00., storeName=004290d2dcb18ca400bd2b34f204aa00/C, priority=9, startTime=1679273826888; duration=0sec 2023-03-20 00:57:07,109 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:07,506 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/1a169689e1ef47599a86d4d77f71aab4 2023-03-20 00:57:07,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:57:07,762 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. as already flushing 2023-03-20 00:57:07,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:07,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52604 deadline: 1679273887772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:07,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:07,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52622 deadline: 1679273887775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:07,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:07,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52574 deadline: 1679273887775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:07,782 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:07,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52558 deadline: 1679273887781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:07,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:07,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52616 deadline: 1679273887784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:07,880 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:07,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52604 deadline: 1679273887875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:07,880 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:07,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52622 deadline: 1679273887876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:07,918 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/2285a2e0abb84fd980f848607068288e 2023-03-20 00:57:07,922 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/3892412c373b479fb9b1d613fc34de29 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/3892412c373b479fb9b1d613fc34de29 2023-03-20 00:57:07,935 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/3892412c373b479fb9b1d613fc34de29, entries=150, sequenceid=256, filesize=25.3 K 2023-03-20 00:57:07,936 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/1a169689e1ef47599a86d4d77f71aab4 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/1a169689e1ef47599a86d4d77f71aab4 2023-03-20 00:57:07,940 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/1a169689e1ef47599a86d4d77f71aab4, entries=150, sequenceid=256, filesize=11.8 K 2023-03-20 00:57:07,941 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/2285a2e0abb84fd980f848607068288e as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/2285a2e0abb84fd980f848607068288e 2023-03-20 00:57:07,944 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/2285a2e0abb84fd980f848607068288e, entries=150, sequenceid=256, filesize=11.8 K 2023-03-20 00:57:07,944 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 004290d2dcb18ca400bd2b34f204aa00 in 1000ms, sequenceid=256, compaction requested=false 2023-03-20 00:57:07,944 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:57:07,945 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:07,945 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:07,945 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:07,945 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:07,945 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:07,945 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:07,945 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:07,945 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:07,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:07,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:07,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:07,946 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:07,946 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:07,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:07,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:07,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:07,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:07,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:07,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:07,948 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:07,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a30b89c[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:07,948 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:07,948 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:07,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:07,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:07,948 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:07,948 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:07,948 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:07,948 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:07,948 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:07,949 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:07,949 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:07,949 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:07,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:07,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:07,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:07,949 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:07,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:07,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:07,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:07,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:07,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:07,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:07,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:07,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:07,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:07,951 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:07,951 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:07,951 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:07,951 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:07,951 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:07,952 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:07,952 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:07,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:07,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:07,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:07,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:07,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:07,952 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:07,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:07,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:07,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:07,952 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:07,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:07,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:07,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:07,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:07,953 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:07,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:07,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:07,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:07,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:07,955 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:07,955 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:07,955 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:07,956 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:07,956 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:07,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:07,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:07,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:07,956 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:07,956 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:07,956 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:07,956 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:07,956 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:07,956 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:07,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:07,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:07,956 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:07,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:07,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:08,056 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:08,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:08,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:08,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:08,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:08,061 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:08,061 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:08,061 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:08,061 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:08,061 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:08,061 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:08,062 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:08,062 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:08,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:08,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:08,062 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:08,062 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:08,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:08,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:08,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:08,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:08,063 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:08,075 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:08,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:08,075 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:08,075 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:08,075 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:08,075 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:08,075 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:08,076 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:08,076 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:57:08,076 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 004290d2dcb18ca400bd2b34f204aa00 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-20 00:57:08,076 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=A 2023-03-20 00:57:08,076 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:08,076 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=B 2023-03-20 00:57:08,076 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:08,076 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=C 2023-03-20 00:57:08,076 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:08,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:57:08,084 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. as already flushing 2023-03-20 00:57:08,106 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032073ea1f905a1b402abf84c99c5e3c6375 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032073ea1f905a1b402abf84c99c5e3c6375 2023-03-20 00:57:08,125 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=271, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/7d706aacb94743e894534325ff77c95d 2023-03-20 00:57:08,149 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/f01b811ee05e4dc8a657bc3595936fa5 2023-03-20 00:57:08,173 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:08,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52622 deadline: 1679273888167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:08,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:08,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52604 deadline: 1679273888168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:08,278 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:08,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52622 deadline: 1679273888274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:08,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:08,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52604 deadline: 1679273888274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:08,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:08,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52604 deadline: 1679273888479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:08,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:08,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52622 deadline: 1679273888480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:08,565 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/60b60672d0e442a882419469c97c77aa 2023-03-20 00:57:08,569 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/7d706aacb94743e894534325ff77c95d as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/7d706aacb94743e894534325ff77c95d 2023-03-20 00:57:08,572 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/7d706aacb94743e894534325ff77c95d, entries=150, sequenceid=271, filesize=25.4 K 2023-03-20 00:57:08,575 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/f01b811ee05e4dc8a657bc3595936fa5 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/f01b811ee05e4dc8a657bc3595936fa5 2023-03-20 00:57:08,579 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/f01b811ee05e4dc8a657bc3595936fa5, entries=150, sequenceid=271, filesize=11.9 K 2023-03-20 00:57:08,580 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/60b60672d0e442a882419469c97c77aa as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/60b60672d0e442a882419469c97c77aa 2023-03-20 00:57:08,585 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/60b60672d0e442a882419469c97c77aa, entries=150, sequenceid=271, filesize=11.9 K 2023-03-20 00:57:08,586 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 004290d2dcb18ca400bd2b34f204aa00 in 510ms, sequenceid=271, compaction requested=true 2023-03-20 00:57:08,586 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:57:08,586 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:08,586 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:08,586 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:08,586 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:08,586 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:08,586 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:08,587 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:08,587 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:08,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:08,589 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:08,589 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:08,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:08,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:08,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:08,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:08,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:08,590 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:08,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@653d9f23[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:08,590 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:08,590 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:08,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:08,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:08,590 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:08,590 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:08,590 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:08,591 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:08,591 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:08,591 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:08,591 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:08,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:08,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:08,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:08,591 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:08,591 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:08,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:08,593 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:08,593 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:08,593 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:08,593 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:08,593 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:08,594 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:08,594 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:08,594 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:08,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:08,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:08,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:08,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:08,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:08,595 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:08,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:08,596 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:08,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:08,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:08,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:08,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:08,598 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:08,598 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:08,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:08,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:08,598 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:08,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:08,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:08,599 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:08,598 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:08,599 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:08,599 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:08,599 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:08,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:08,599 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:08,599 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:08,599 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:08,599 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:08,699 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:08,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:08,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:08,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:08,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:08,701 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:08,701 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:08,701 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:08,701 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:08,702 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:08,702 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:08,702 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:08,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:08,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:08,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:08,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:08,703 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:08,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:08,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:08,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:08,704 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:08,704 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:08,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:08,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:08,706 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:08,706 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:08,706 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:08,710 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:08,710 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:08,710 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:08,710 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:57:08,710 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 004290d2dcb18ca400bd2b34f204aa00 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-20 00:57:08,710 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=A 2023-03-20 00:57:08,710 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:08,711 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=B 2023-03-20 00:57:08,711 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:08,711 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=C 2023-03-20 00:57:08,711 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:08,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:57:08,785 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. as already flushing 2023-03-20 00:57:08,822 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:08,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52604 deadline: 1679273888817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:08,833 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:08,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52622 deadline: 1679273888823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:08,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:08,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52604 deadline: 1679273888923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:08,943 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:08,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52622 deadline: 1679273888935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:09,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:09,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52604 deadline: 1679273889129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:09,147 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:09,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52622 deadline: 1679273889145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:09,161 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032013c5302ba043490c90f26715b359353f to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032013c5302ba043490c90f26715b359353f 2023-03-20 00:57:09,437 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:09,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52604 deadline: 1679273889433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:09,451 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:09,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52622 deadline: 1679273889448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:09,566 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=291, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/a9c24beab8c04af88b8ece360cd99353 2023-03-20 00:57:09,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:09,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52574 deadline: 1679273889782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:09,786 DEBUG [Thread-2059] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4153 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00., hostname=jenkins-hbase20.apache.org,34553,1679273725199, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 00:57:09,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:09,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52558 deadline: 1679273889786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:09,790 DEBUG [Thread-2057] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4157 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00., hostname=jenkins-hbase20.apache.org,34553,1679273725199, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 00:57:09,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:09,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52616 deadline: 1679273889800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:09,805 DEBUG [Thread-2063] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4169 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00., hostname=jenkins-hbase20.apache.org,34553,1679273725199, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 00:57:09,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:09,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52604 deadline: 1679273889938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:09,955 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:09,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52622 deadline: 1679273889955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:09,974 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/528a19338a9b480fa0793f36e8aac903 2023-03-20 00:57:10,382 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/e3e12848e1e64899a00cffc6dad6bb8c 2023-03-20 00:57:10,385 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/a9c24beab8c04af88b8ece360cd99353 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/a9c24beab8c04af88b8ece360cd99353 2023-03-20 00:57:10,388 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/a9c24beab8c04af88b8ece360cd99353, entries=150, sequenceid=291, filesize=25.4 K 2023-03-20 00:57:10,389 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/528a19338a9b480fa0793f36e8aac903 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/528a19338a9b480fa0793f36e8aac903 2023-03-20 00:57:10,398 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/528a19338a9b480fa0793f36e8aac903, entries=150, sequenceid=291, filesize=11.9 K 2023-03-20 00:57:10,398 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/e3e12848e1e64899a00cffc6dad6bb8c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/e3e12848e1e64899a00cffc6dad6bb8c 2023-03-20 00:57:10,408 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/e3e12848e1e64899a00cffc6dad6bb8c, entries=150, sequenceid=291, filesize=11.9 K 2023-03-20 00:57:10,409 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 004290d2dcb18ca400bd2b34f204aa00 in 1699ms, sequenceid=291, compaction requested=true 2023-03-20 00:57:10,409 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:57:10,409 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:10,409 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:10,409 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:10,409 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:10,409 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:10,409 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:10,410 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:10,410 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:10,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:10,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:10,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:10,411 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:10,411 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:10,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:10,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:10,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:10,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:10,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:10,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:10,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70b11a2f[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:10,412 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:10,412 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:10,413 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:10,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:10,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:10,413 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:10,413 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:10,413 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:10,413 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:10,413 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:10,414 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:10,414 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:10,414 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:10,414 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:10,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:10,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:10,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:10,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:10,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:10,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:10,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:10,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:10,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:10,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:10,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:10,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:10,416 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:10,416 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:10,416 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:10,416 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:10,416 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:10,417 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:10,417 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:10,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:10,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:10,417 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:10,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:10,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:10,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:10,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:10,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:10,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:10,418 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:10,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:10,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:10,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:10,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:10,420 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:10,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:10,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:10,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:10,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:10,421 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:10,421 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:10,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:10,421 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:10,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:10,421 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:10,421 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:10,421 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:10,421 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:10,421 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:10,421 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:10,421 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:10,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:10,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:10,422 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:10,422 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:10,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:10,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:10,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:10,522 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:10,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:10,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:10,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:10,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:10,527 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:10,527 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:10,527 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:10,527 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:10,527 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:10,527 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:10,528 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:10,528 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:10,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:10,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:10,528 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:10,528 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:10,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:10,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:10,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:10,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:10,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:10,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:10,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:10,529 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:10,529 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:10,529 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:10,529 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:10,529 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:10,529 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:10,529 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:57:10,529 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 004290d2dcb18ca400bd2b34f204aa00 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-20 00:57:10,529 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=A 2023-03-20 00:57:10,530 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:10,530 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=B 2023-03-20 00:57:10,530 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:10,530 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=C 2023-03-20 00:57:10,530 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:10,560 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320b872bc4f3411419794d3a45c254dfb86 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320b872bc4f3411419794d3a45c254dfb86 2023-03-20 00:57:10,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:57:10,950 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. as already flushing 2023-03-20 00:57:10,965 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=303, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/2e470822b06c4c9a9f5024b3b8aed214 2023-03-20 00:57:11,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:11,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52622 deadline: 1679273891013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:11,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:11,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52604 deadline: 1679273891017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:11,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:11,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52622 deadline: 1679273891117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:11,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:11,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52604 deadline: 1679273891124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:11,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:11,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52622 deadline: 1679273891324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:11,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:11,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52604 deadline: 1679273891330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:11,374 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/189cf4fb5358449f8d085ef590fbf9e2 2023-03-20 00:57:11,386 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/55d8177ce1a3467780e648ddae17e2fa 2023-03-20 00:57:11,401 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/2e470822b06c4c9a9f5024b3b8aed214 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/2e470822b06c4c9a9f5024b3b8aed214 2023-03-20 00:57:11,407 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/2e470822b06c4c9a9f5024b3b8aed214, entries=150, sequenceid=303, filesize=25.4 K 2023-03-20 00:57:11,408 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/189cf4fb5358449f8d085ef590fbf9e2 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/189cf4fb5358449f8d085ef590fbf9e2 2023-03-20 00:57:11,412 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/189cf4fb5358449f8d085ef590fbf9e2, entries=150, sequenceid=303, filesize=11.9 K 2023-03-20 00:57:11,413 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/55d8177ce1a3467780e648ddae17e2fa as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/55d8177ce1a3467780e648ddae17e2fa 2023-03-20 00:57:11,423 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/55d8177ce1a3467780e648ddae17e2fa, entries=150, sequenceid=303, filesize=11.9 K 2023-03-20 00:57:11,424 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 004290d2dcb18ca400bd2b34f204aa00 in 895ms, sequenceid=303, compaction requested=true 2023-03-20 00:57:11,424 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:57:11,424 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:11,424 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:11,424 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:11,424 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:11,424 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:11,424 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:11,425 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:11,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:11,425 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:11,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:11,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:11,426 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:11,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:11,426 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:11,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:11,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:11,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:11,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:11,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:11,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b6dd7e3[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:11,427 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:11,427 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:11,428 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:11,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:11,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:11,428 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:11,428 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:11,428 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:11,429 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:11,429 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:11,429 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:11,429 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:11,429 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:11,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:11,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:11,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:11,429 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:11,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:11,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:11,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:11,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:11,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:11,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:11,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:11,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:11,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:11,431 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:11,431 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:11,432 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:11,432 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:11,432 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:11,433 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:11,433 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:11,433 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:11,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:11,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:11,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:11,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:11,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:11,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:11,433 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:11,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:11,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:11,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:11,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:11,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:11,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:11,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:11,437 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:11,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:11,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:11,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:11,439 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:11,439 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:11,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:11,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:11,439 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:11,439 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:11,439 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:11,439 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:11,439 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:11,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:11,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:11,439 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:11,439 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:11,439 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:11,439 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:11,439 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:11,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:11,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:11,540 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:11,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:11,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:11,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:11,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:11,554 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:11,554 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:11,554 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:11,554 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:11,554 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:11,554 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:11,557 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:11,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:11,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:11,557 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:11,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:11,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:11,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:11,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:11,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:11,560 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:11,560 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:11,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:11,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:11,562 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:11,562 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:11,562 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:11,569 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:11,569 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:11,569 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:11,569 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. started... 2023-03-20 00:57:11,569 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 004290d2dcb18ca400bd2b34f204aa00 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-20 00:57:11,569 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=A 2023-03-20 00:57:11,570 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:11,586 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=B 2023-03-20 00:57:11,586 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:11,586 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=C 2023-03-20 00:57:11,586 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:11,635 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320978c1972ca9a421c8339a25f023113dd to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320978c1972ca9a421c8339a25f023113dd 2023-03-20 00:57:11,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:57:11,640 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. as already flushing 2023-03-20 00:57:11,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:11,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52622 deadline: 1679273891673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:11,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:11,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52604 deadline: 1679273891673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:11,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:11,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52622 deadline: 1679273891782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:11,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:11,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52604 deadline: 1679273891782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:11,997 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:11,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52622 deadline: 1679273891988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:11,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:11,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52604 deadline: 1679273891988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:12,035 DEBUG [Thread-2068] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5144fcff to 127.0.0.1:60547 2023-03-20 00:57:12,035 DEBUG [Thread-2068] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:57:12,037 DEBUG [Thread-2074] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4af630f2 to 127.0.0.1:60547 2023-03-20 00:57:12,037 DEBUG [Thread-2074] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:57:12,040 DEBUG [Thread-2066] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ba3ce53 to 127.0.0.1:60547 2023-03-20 00:57:12,040 DEBUG [Thread-2066] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:57:12,041 DEBUG [Thread-2070] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x54da4cc4 to 127.0.0.1:60547 2023-03-20 00:57:12,041 DEBUG [Thread-2070] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:57:12,043 DEBUG [Thread-2072] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1280c068 to 127.0.0.1:60547 2023-03-20 00:57:12,043 DEBUG [Thread-2072] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:57:12,054 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=323, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/bf31c5a9266f420c954c23695d5b92ea 2023-03-20 00:57:12,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:12,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52622 deadline: 1679273892299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:12,300 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:12,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52604 deadline: 1679273892300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:12,463 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=323 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/fec30000a3d54b1f9785d618d06bf637 2023-03-20 00:57:12,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:12,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52622 deadline: 1679273892801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:12,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:12,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52604 deadline: 1679273892803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:12,873 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=323 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/3d7161bee0934e79a0524aff5ee27b99 2023-03-20 00:57:12,877 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/bf31c5a9266f420c954c23695d5b92ea as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/bf31c5a9266f420c954c23695d5b92ea 2023-03-20 00:57:12,880 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/bf31c5a9266f420c954c23695d5b92ea, entries=150, sequenceid=323, filesize=25.4 K 2023-03-20 00:57:12,880 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/fec30000a3d54b1f9785d618d06bf637 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/fec30000a3d54b1f9785d618d06bf637 2023-03-20 00:57:12,884 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/fec30000a3d54b1f9785d618d06bf637, entries=150, sequenceid=323, filesize=11.9 K 2023-03-20 00:57:12,885 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/3d7161bee0934e79a0524aff5ee27b99 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/3d7161bee0934e79a0524aff5ee27b99 2023-03-20 00:57:12,888 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/3d7161bee0934e79a0524aff5ee27b99, entries=150, sequenceid=323, filesize=11.9 K 2023-03-20 00:57:12,889 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 004290d2dcb18ca400bd2b34f204aa00 in 1320ms, sequenceid=323, compaction requested=true 2023-03-20 00:57:12,889 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:57:12,889 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:12,889 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:12,889 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:12,889 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:12,889 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:12,889 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:12,890 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:12,890 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:12,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:12,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:12,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:12,890 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:12,890 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:12,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:12,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:12,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:12,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:12,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:12,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:12,891 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:12,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64467c4a[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:12,891 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:12,892 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:12,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:12,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:12,892 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:12,892 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:12,892 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:12,892 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:12,892 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:12,893 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:12,893 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:12,893 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:12,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:12,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:12,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:12,893 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:12,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:12,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:12,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:12,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:12,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:12,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:12,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:12,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:12,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:12,894 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:12,894 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:12,894 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:12,894 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:12,894 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:12,895 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:12,895 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:12,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:12,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:12,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:12,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:12,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:12,895 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:12,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:12,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:12,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:12,896 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:12,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:12,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:12,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:12,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:12,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:12,897 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:12,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:12,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:12,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:12,898 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:12,898 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:12,898 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:12,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:12,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:12,899 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:12,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:12,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:12,899 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:12,899 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:12,899 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:12,899 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:12,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:12,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:12,899 DEBUG [Thread-2065] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:12,899 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:12,899 DEBUG [Thread-2065] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:12,899 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:12,999 DEBUG [Thread-2065] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:13,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:13,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:57:13,792 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 004290d2dcb18ca400bd2b34f204aa00 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-20 00:57:13,792 DEBUG [Thread-2057] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d80e5a8 to 127.0.0.1:60547 2023-03-20 00:57:13,792 DEBUG [Thread-2059] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ea1a2c1 to 127.0.0.1:60547 2023-03-20 00:57:13,792 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=A 2023-03-20 00:57:13,792 DEBUG [Thread-2057] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:57:13,793 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:13,792 DEBUG [Thread-2059] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:57:13,793 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=B 2023-03-20 00:57:13,793 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:13,793 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=C 2023-03-20 00:57:13,793 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:13,806 DEBUG [Thread-2055] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d63a8ce to 127.0.0.1:60547 2023-03-20 00:57:13,806 DEBUG [Thread-2055] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:57:13,811 DEBUG [Thread-2061] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x25185572 to 127.0.0.1:60547 2023-03-20 00:57:13,811 DEBUG [Thread-2061] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:57:13,840 DEBUG [Thread-2063] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x35f0f398 to 127.0.0.1:60547 2023-03-20 00:57:13,840 DEBUG [Thread-2063] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:57:13,840 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-20 00:57:13,840 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(394): wrote 72 2023-03-20 00:57:13,840 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(394): wrote 44 2023-03-20 00:57:13,840 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(394): wrote 41 2023-03-20 00:57:13,840 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(394): wrote 76 2023-03-20 00:57:13,840 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(394): wrote 41 2023-03-20 00:57:13,840 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-20 00:57:13,840 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-20 00:57:13,840 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(402): scanned 1871 2023-03-20 00:57:13,841 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(403): verified 5611 rows 2023-03-20 00:57:13,841 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(402): scanned 1854 2023-03-20 00:57:13,841 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(403): verified 5560 rows 2023-03-20 00:57:13,841 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(402): scanned 1861 2023-03-20 00:57:13,841 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(403): verified 5578 rows 2023-03-20 00:57:13,841 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(402): scanned 1844 2023-03-20 00:57:13,841 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(403): verified 5530 rows 2023-03-20 00:57:13,841 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(402): scanned 1850 2023-03-20 00:57:13,841 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(403): verified 5548 rows 2023-03-20 00:57:13,841 INFO [Listener at localhost.localdomain/43361] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-03-20 00:57:13,841 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1eaa86ba to 127.0.0.1:60547 2023-03-20 00:57:13,841 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:57:13,843 INFO [Listener at localhost.localdomain/43361] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-20 00:57:13,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-03-20 00:57:13,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-20 00:57:13,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-20 00:57:13,853 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679273833853"}]},"ts":"1679273833853"} 2023-03-20 00:57:13,860 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-20 00:57:13,896 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-20 00:57:13,897 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=004290d2dcb18ca400bd2b34f204aa00, UNASSIGN}] 2023-03-20 00:57:13,897 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=004290d2dcb18ca400bd2b34f204aa00, UNASSIGN 2023-03-20 00:57:13,899 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=004290d2dcb18ca400bd2b34f204aa00, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:13,899 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273833899"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679273833899"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679273833899"}]},"ts":"1679273833899"} 2023-03-20 00:57:13,900 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199}] 2023-03-20 00:57:13,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-20 00:57:14,052 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:57:14,052 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 004290d2dcb18ca400bd2b34f204aa00, disabling compactions & flushes 2023-03-20 00:57:14,052 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:14,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-20 00:57:14,202 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303208423bbc8fd7644be83dba768ae278a38 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303208423bbc8fd7644be83dba768ae278a38 2023-03-20 00:57:14,212 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=337, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/8374d000ce494fddb2ffae1d0b708a28 2023-03-20 00:57:14,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-20 00:57:14,632 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/b8da6f3dff01497382e4497d06366da9 2023-03-20 00:57:14,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-20 00:57:15,041 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/80687481b8aa4559a23049cff1fab509 2023-03-20 00:57:15,051 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/8374d000ce494fddb2ffae1d0b708a28 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/8374d000ce494fddb2ffae1d0b708a28 2023-03-20 00:57:15,059 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/8374d000ce494fddb2ffae1d0b708a28, entries=150, sequenceid=337, filesize=25.4 K 2023-03-20 00:57:15,060 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/b8da6f3dff01497382e4497d06366da9 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/b8da6f3dff01497382e4497d06366da9 2023-03-20 00:57:15,064 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/b8da6f3dff01497382e4497d06366da9, entries=150, sequenceid=337, filesize=11.9 K 2023-03-20 00:57:15,065 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/80687481b8aa4559a23049cff1fab509 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/80687481b8aa4559a23049cff1fab509 2023-03-20 00:57:15,070 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/80687481b8aa4559a23049cff1fab509, entries=150, sequenceid=337, filesize=11.9 K 2023-03-20 00:57:15,070 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=20.13 KB/20610 for 004290d2dcb18ca400bd2b34f204aa00 in 1278ms, sequenceid=337, compaction requested=true 2023-03-20 00:57:15,070 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:57:15,070 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:15,071 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:15,071 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. because compaction request was cancelled 2023-03-20 00:57:15,071 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:15,071 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. after waiting 0 ms 2023-03-20 00:57:15,071 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:15,071 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. because compaction request was cancelled 2023-03-20 00:57:15,071 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 004290d2dcb18ca400bd2b34f204aa00 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-20 00:57:15,071 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:15,071 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:15,071 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. because compaction request was cancelled 2023-03-20 00:57:15,071 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=A 2023-03-20 00:57:15,071 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:15,071 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=B 2023-03-20 00:57:15,071 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:15,071 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 004290d2dcb18ca400bd2b34f204aa00, store=C 2023-03-20 00:57:15,072 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:15,484 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303209beaf2f4f4424f1f904675a5c112998e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303209beaf2f4f4424f1f904675a5c112998e 2023-03-20 00:57:15,889 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=343, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/8220cecc1da54854b48386d5dfeb98dc 2023-03-20 00:57:15,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-20 00:57:16,297 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=343 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/deac310eba11409eac81fb63fefde593 2023-03-20 00:57:16,705 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=343 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/ef5d1e53c52f446292a510106b4230f9 2023-03-20 00:57:16,709 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/A/8220cecc1da54854b48386d5dfeb98dc as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/8220cecc1da54854b48386d5dfeb98dc 2023-03-20 00:57:16,712 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/8220cecc1da54854b48386d5dfeb98dc, entries=150, sequenceid=343, filesize=25.4 K 2023-03-20 00:57:16,713 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/B/deac310eba11409eac81fb63fefde593 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/deac310eba11409eac81fb63fefde593 2023-03-20 00:57:16,717 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/deac310eba11409eac81fb63fefde593, entries=150, sequenceid=343, filesize=11.9 K 2023-03-20 00:57:16,717 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/.tmp/C/ef5d1e53c52f446292a510106b4230f9 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/ef5d1e53c52f446292a510106b4230f9 2023-03-20 00:57:16,720 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/ef5d1e53c52f446292a510106b4230f9, entries=150, sequenceid=343, filesize=11.9 K 2023-03-20 00:57:16,721 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 004290d2dcb18ca400bd2b34f204aa00 in 1650ms, sequenceid=343, compaction requested=true 2023-03-20 00:57:16,721 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/4cdeb9f033484d0190c426529625eb20, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/1655bf332ed04b8db4c856c2a4a57622, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/8ed0cd9329334e48890d0011399069e7, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/e6ae9899334f4b70a593429c5cd1eabe, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/8c4159f4438046ca8fc3425d8b43c96d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/f6aa592362b249a0bebc590d37b4e04d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/eb96605b9a554661875758c80ff9c654, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/d79a1aedb27e469f9271be82ea0f16a7, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/78a3228a3011403a9208a613a18f2a9d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/df0bfe298b7948a38127bb9b546cae5b, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/eb383bf3d20e488494035bcf12f462a2, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/3340dd188c72466cb83dca8a34c47466, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/4fa2998a65084e11b334283720c2d13e] to archive 2023-03-20 00:57:16,722 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 00:57:16,723 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/4cdeb9f033484d0190c426529625eb20 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/4cdeb9f033484d0190c426529625eb20 2023-03-20 00:57:16,724 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/1655bf332ed04b8db4c856c2a4a57622 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/1655bf332ed04b8db4c856c2a4a57622 2023-03-20 00:57:16,725 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/8ed0cd9329334e48890d0011399069e7 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/8ed0cd9329334e48890d0011399069e7 2023-03-20 00:57:16,726 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/e6ae9899334f4b70a593429c5cd1eabe to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/e6ae9899334f4b70a593429c5cd1eabe 2023-03-20 00:57:16,727 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/8c4159f4438046ca8fc3425d8b43c96d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/8c4159f4438046ca8fc3425d8b43c96d 2023-03-20 00:57:16,728 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/f6aa592362b249a0bebc590d37b4e04d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/f6aa592362b249a0bebc590d37b4e04d 2023-03-20 00:57:16,729 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/eb96605b9a554661875758c80ff9c654 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/eb96605b9a554661875758c80ff9c654 2023-03-20 00:57:16,730 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/d79a1aedb27e469f9271be82ea0f16a7 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/d79a1aedb27e469f9271be82ea0f16a7 2023-03-20 00:57:16,730 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/78a3228a3011403a9208a613a18f2a9d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/78a3228a3011403a9208a613a18f2a9d 2023-03-20 00:57:16,731 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/df0bfe298b7948a38127bb9b546cae5b to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/df0bfe298b7948a38127bb9b546cae5b 2023-03-20 00:57:16,732 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/eb383bf3d20e488494035bcf12f462a2 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/eb383bf3d20e488494035bcf12f462a2 2023-03-20 00:57:16,733 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/3340dd188c72466cb83dca8a34c47466 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/3340dd188c72466cb83dca8a34c47466 2023-03-20 00:57:16,734 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/4fa2998a65084e11b334283720c2d13e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/4fa2998a65084e11b334283720c2d13e 2023-03-20 00:57:16,736 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/6a490f891a7645d3bcb7d272eae79f6e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/203d0f0795724ebd9bab5ac860413fba, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/b80a3bea4980411fa9dccbf664ba06fa, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/d48830c254424f5683aaa9d04e55cf88, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/123a1b7e532e43dcb672a4c7ad604651, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/fae381505cb94dbfa0c1c871dfdc2365, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/61b36a3baa3a4dcdb3cb71477c10722b, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/00c6ebd6f92a48608db87453ce5f15f1, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/372d7b6991a2480e8653e2a4046a2af4, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/537596fdf29f4018bea10e4d1f08c1ea, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/f7bf2dc3a19048388dd4cab3101fafd9, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/0a95a3d0c87c4f3e9b448b5c979f9b9d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/793737a205dc44e182d9b5fad59a64d2] to archive 2023-03-20 00:57:16,736 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 00:57:16,737 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/6a490f891a7645d3bcb7d272eae79f6e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/6a490f891a7645d3bcb7d272eae79f6e 2023-03-20 00:57:16,738 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/203d0f0795724ebd9bab5ac860413fba to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/203d0f0795724ebd9bab5ac860413fba 2023-03-20 00:57:16,739 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/b80a3bea4980411fa9dccbf664ba06fa to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/b80a3bea4980411fa9dccbf664ba06fa 2023-03-20 00:57:16,740 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/d48830c254424f5683aaa9d04e55cf88 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/d48830c254424f5683aaa9d04e55cf88 2023-03-20 00:57:16,741 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/123a1b7e532e43dcb672a4c7ad604651 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/123a1b7e532e43dcb672a4c7ad604651 2023-03-20 00:57:16,742 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/fae381505cb94dbfa0c1c871dfdc2365 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/fae381505cb94dbfa0c1c871dfdc2365 2023-03-20 00:57:16,742 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/61b36a3baa3a4dcdb3cb71477c10722b to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/61b36a3baa3a4dcdb3cb71477c10722b 2023-03-20 00:57:16,743 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/00c6ebd6f92a48608db87453ce5f15f1 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/00c6ebd6f92a48608db87453ce5f15f1 2023-03-20 00:57:16,744 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/372d7b6991a2480e8653e2a4046a2af4 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/372d7b6991a2480e8653e2a4046a2af4 2023-03-20 00:57:16,744 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/537596fdf29f4018bea10e4d1f08c1ea to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/537596fdf29f4018bea10e4d1f08c1ea 2023-03-20 00:57:16,745 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/f7bf2dc3a19048388dd4cab3101fafd9 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/f7bf2dc3a19048388dd4cab3101fafd9 2023-03-20 00:57:16,746 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/0a95a3d0c87c4f3e9b448b5c979f9b9d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/0a95a3d0c87c4f3e9b448b5c979f9b9d 2023-03-20 00:57:16,746 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/793737a205dc44e182d9b5fad59a64d2 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/793737a205dc44e182d9b5fad59a64d2 2023-03-20 00:57:16,748 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/b4c7b3e263f646abb3bd5a6ccb055bef, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/0a419803d94840b796bd17d6a04cdcb4, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/76e5a42f11f74020b5720d1f554ddc1d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/b6cfc6960e924c1c8aaa1bc9b5df705c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/85912b4c9d474acbaac1433c16258d63, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/209296d3e74c410bac399e60283ac1af, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/80fbcc5f8c5d477193cd9607ba6b13d2, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/a81c15cbac2f4279868bb0dea0237396, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/7ba7fc30ec3b44e8b3201247d5c9330f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/3638f581119c4358af88cf5a35868316, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/71c4b52b06934cc288208a02013212c8, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/f13dc0dd4fde4820b97f61a5a6785b74, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/ea101ef110d449b19a94234487dbf4ea] to archive 2023-03-20 00:57:16,749 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 00:57:16,750 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/b4c7b3e263f646abb3bd5a6ccb055bef to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/b4c7b3e263f646abb3bd5a6ccb055bef 2023-03-20 00:57:16,751 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/0a419803d94840b796bd17d6a04cdcb4 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/0a419803d94840b796bd17d6a04cdcb4 2023-03-20 00:57:16,751 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/76e5a42f11f74020b5720d1f554ddc1d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/76e5a42f11f74020b5720d1f554ddc1d 2023-03-20 00:57:16,752 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/b6cfc6960e924c1c8aaa1bc9b5df705c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/b6cfc6960e924c1c8aaa1bc9b5df705c 2023-03-20 00:57:16,753 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/85912b4c9d474acbaac1433c16258d63 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/85912b4c9d474acbaac1433c16258d63 2023-03-20 00:57:16,753 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/209296d3e74c410bac399e60283ac1af to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/209296d3e74c410bac399e60283ac1af 2023-03-20 00:57:16,754 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/80fbcc5f8c5d477193cd9607ba6b13d2 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/80fbcc5f8c5d477193cd9607ba6b13d2 2023-03-20 00:57:16,755 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/a81c15cbac2f4279868bb0dea0237396 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/a81c15cbac2f4279868bb0dea0237396 2023-03-20 00:57:16,755 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/7ba7fc30ec3b44e8b3201247d5c9330f to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/7ba7fc30ec3b44e8b3201247d5c9330f 2023-03-20 00:57:16,756 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/3638f581119c4358af88cf5a35868316 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/3638f581119c4358af88cf5a35868316 2023-03-20 00:57:16,757 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/71c4b52b06934cc288208a02013212c8 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/71c4b52b06934cc288208a02013212c8 2023-03-20 00:57:16,757 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/f13dc0dd4fde4820b97f61a5a6785b74 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/f13dc0dd4fde4820b97f61a5a6785b74 2023-03-20 00:57:16,758 DEBUG [StoreCloser-TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/ea101ef110d449b19a94234487dbf4ea to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/ea101ef110d449b19a94234487dbf4ea 2023-03-20 00:57:16,762 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/recovered.edits/346.seqid, newMaxSeqId=346, maxSeqId=4 2023-03-20 00:57:16,763 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00. 2023-03-20 00:57:16,763 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 004290d2dcb18ca400bd2b34f204aa00: 2023-03-20 00:57:16,764 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:57:16,764 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=004290d2dcb18ca400bd2b34f204aa00, regionState=CLOSED 2023-03-20 00:57:16,764 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273836764"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679273836764"}]},"ts":"1679273836764"} 2023-03-20 00:57:16,767 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-03-20 00:57:16,767 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 004290d2dcb18ca400bd2b34f204aa00, server=jenkins-hbase20.apache.org,34553,1679273725199 in 2.8660 sec 2023-03-20 00:57:16,768 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-03-20 00:57:16,768 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=004290d2dcb18ca400bd2b34f204aa00, UNASSIGN in 2.8700 sec 2023-03-20 00:57:16,768 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679273836768"}]},"ts":"1679273836768"} 2023-03-20 00:57:16,769 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-20 00:57:16,770 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-20 00:57:16,772 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.9280 sec 2023-03-20 00:57:17,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-20 00:57:17,955 INFO [Listener at localhost.localdomain/43361] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-03-20 00:57:17,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-03-20 00:57:17,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:57:17,957 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:57:17,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-20 00:57:17,958 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:57:17,962 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:57:17,964 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A, FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B, FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C, FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/recovered.edits] 2023-03-20 00:57:17,967 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/2e470822b06c4c9a9f5024b3b8aed214 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/2e470822b06c4c9a9f5024b3b8aed214 2023-03-20 00:57:17,968 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/3892412c373b479fb9b1d613fc34de29 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/3892412c373b479fb9b1d613fc34de29 2023-03-20 00:57:17,969 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/7d706aacb94743e894534325ff77c95d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/7d706aacb94743e894534325ff77c95d 2023-03-20 00:57:17,970 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/8220cecc1da54854b48386d5dfeb98dc to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/8220cecc1da54854b48386d5dfeb98dc 2023-03-20 00:57:17,971 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/8374d000ce494fddb2ffae1d0b708a28 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/8374d000ce494fddb2ffae1d0b708a28 2023-03-20 00:57:17,973 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/a9c24beab8c04af88b8ece360cd99353 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/a9c24beab8c04af88b8ece360cd99353 2023-03-20 00:57:17,974 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/bf31c5a9266f420c954c23695d5b92ea to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/bf31c5a9266f420c954c23695d5b92ea 2023-03-20 00:57:17,975 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/c4dd0adb5f6a429fb0bb2fb9bb283478 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/A/c4dd0adb5f6a429fb0bb2fb9bb283478 2023-03-20 00:57:17,977 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/189cf4fb5358449f8d085ef590fbf9e2 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/189cf4fb5358449f8d085ef590fbf9e2 2023-03-20 00:57:17,978 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/1a169689e1ef47599a86d4d77f71aab4 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/1a169689e1ef47599a86d4d77f71aab4 2023-03-20 00:57:17,979 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/528a19338a9b480fa0793f36e8aac903 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/528a19338a9b480fa0793f36e8aac903 2023-03-20 00:57:17,980 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/9ca1eb4b32f24abbb42ae7564f8e0c1d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/9ca1eb4b32f24abbb42ae7564f8e0c1d 2023-03-20 00:57:17,981 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/b8da6f3dff01497382e4497d06366da9 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/b8da6f3dff01497382e4497d06366da9 2023-03-20 00:57:17,982 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/deac310eba11409eac81fb63fefde593 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/deac310eba11409eac81fb63fefde593 2023-03-20 00:57:17,983 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/f01b811ee05e4dc8a657bc3595936fa5 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/f01b811ee05e4dc8a657bc3595936fa5 2023-03-20 00:57:17,984 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/fec30000a3d54b1f9785d618d06bf637 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/B/fec30000a3d54b1f9785d618d06bf637 2023-03-20 00:57:17,987 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/2285a2e0abb84fd980f848607068288e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/2285a2e0abb84fd980f848607068288e 2023-03-20 00:57:17,987 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/3d7161bee0934e79a0524aff5ee27b99 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/3d7161bee0934e79a0524aff5ee27b99 2023-03-20 00:57:17,988 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/55d8177ce1a3467780e648ddae17e2fa to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/55d8177ce1a3467780e648ddae17e2fa 2023-03-20 00:57:17,990 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/60b60672d0e442a882419469c97c77aa to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/60b60672d0e442a882419469c97c77aa 2023-03-20 00:57:17,991 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/80687481b8aa4559a23049cff1fab509 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/80687481b8aa4559a23049cff1fab509 2023-03-20 00:57:17,995 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/cdc4669909e448e78721efe4bf31e3ed to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/cdc4669909e448e78721efe4bf31e3ed 2023-03-20 00:57:17,997 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/e3e12848e1e64899a00cffc6dad6bb8c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/e3e12848e1e64899a00cffc6dad6bb8c 2023-03-20 00:57:17,998 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/ef5d1e53c52f446292a510106b4230f9 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/C/ef5d1e53c52f446292a510106b4230f9 2023-03-20 00:57:18,001 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/recovered.edits/346.seqid to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00/recovered.edits/346.seqid 2023-03-20 00:57:18,002 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/004290d2dcb18ca400bd2b34f204aa00 2023-03-20 00:57:18,002 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-20 00:57:18,002 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-20 00:57:18,003 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-20 00:57:18,006 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032013c5302ba043490c90f26715b359353f to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032013c5302ba043490c90f26715b359353f 2023-03-20 00:57:18,008 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032014233b6cb80d40fcb00943f42d6270f6 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032014233b6cb80d40fcb00943f42d6270f6 2023-03-20 00:57:18,009 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303202017cab7676b48d79c63135cffa10e9c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303202017cab7676b48d79c63135cffa10e9c 2023-03-20 00:57:18,011 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032020bc383c40aa4daf8ec2c010ca984794 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032020bc383c40aa4daf8ec2c010ca984794 2023-03-20 00:57:18,012 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320477ca1cb6f28439b9e5cab1ba4e9899b to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320477ca1cb6f28439b9e5cab1ba4e9899b 2023-03-20 00:57:18,014 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303205d85075a63444a7d98add88dcf35e3a1 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303205d85075a63444a7d98add88dcf35e3a1 2023-03-20 00:57:18,015 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032073ea1f905a1b402abf84c99c5e3c6375 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032073ea1f905a1b402abf84c99c5e3c6375 2023-03-20 00:57:18,016 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303208423bbc8fd7644be83dba768ae278a38 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303208423bbc8fd7644be83dba768ae278a38 2023-03-20 00:57:18,017 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303208882375595c74ae083569fb0c4667818 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303208882375595c74ae083569fb0c4667818 2023-03-20 00:57:18,018 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320978c1972ca9a421c8339a25f023113dd to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320978c1972ca9a421c8339a25f023113dd 2023-03-20 00:57:18,019 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303209a0313ec358643e693b905b05e4a0efd to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303209a0313ec358643e693b905b05e4a0efd 2023-03-20 00:57:18,020 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303209beaf2f4f4424f1f904675a5c112998e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303209beaf2f4f4424f1f904675a5c112998e 2023-03-20 00:57:18,020 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320b7e53b3643804374a3dbc1f297265e8a to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320b7e53b3643804374a3dbc1f297265e8a 2023-03-20 00:57:18,021 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320b872bc4f3411419794d3a45c254dfb86 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320b872bc4f3411419794d3a45c254dfb86 2023-03-20 00:57:18,022 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320c0bdcb94dfb546c09caade61406137f4 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320c0bdcb94dfb546c09caade61406137f4 2023-03-20 00:57:18,023 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320d3c01fed1af94425a958d077a0cfe6cc to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320d3c01fed1af94425a958d077a0cfe6cc 2023-03-20 00:57:18,024 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320d6aa121ef4f1432e8b1e0a8d13663962 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320d6aa121ef4f1432e8b1e0a8d13663962 2023-03-20 00:57:18,024 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320e049e012efa1434cb2ed1d276d27f7b8 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320e049e012efa1434cb2ed1d276d27f7b8 2023-03-20 00:57:18,025 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320fb0bd447b2bf4296883906b7f8073db0 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320fb0bd447b2bf4296883906b7f8073db0 2023-03-20 00:57:18,026 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-20 00:57:18,027 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:57:18,028 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-20 00:57:18,029 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-20 00:57:18,030 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:57:18,030 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-20 00:57:18,030 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679273838030"}]},"ts":"9223372036854775807"} 2023-03-20 00:57:18,032 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-20 00:57:18,032 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 004290d2dcb18ca400bd2b34f204aa00, NAME => 'TestAcidGuarantees,,1679273808896.004290d2dcb18ca400bd2b34f204aa00.', STARTKEY => '', ENDKEY => ''}] 2023-03-20 00:57:18,032 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-20 00:57:18,032 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679273838032"}]},"ts":"9223372036854775807"} 2023-03-20 00:57:18,033 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-20 00:57:18,034 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:57:18,034 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 79 msec 2023-03-20 00:57:18,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-20 00:57:18,059 INFO [Listener at localhost.localdomain/43361] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-03-20 00:57:18,067 INFO [Listener at localhost.localdomain/43361] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227 (was 232), OpenFileDescriptor=531 (was 546), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=594 (was 540) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=4988 (was 4258) - AvailableMemoryMB LEAK? - 2023-03-20 00:57:18,078 INFO [Listener at localhost.localdomain/43361] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227, OpenFileDescriptor=531, MaxFileDescriptor=60000, SystemLoadAverage=594, ProcessCount=168, AvailableMemoryMB=4987 2023-03-20 00:57:18,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-20 00:57:18,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-20 00:57:18,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-20 00:57:18,082 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-20 00:57:18,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-03-20 00:57:18,083 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-20 00:57:18,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-20 00:57:18,084 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:18,084 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f empty. 2023-03-20 00:57:18,084 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:18,084 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-20 00:57:18,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-20 00:57:18,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-20 00:57:18,492 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-20 00:57:18,494 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 718cd5d8d8fa4a8a2d92da64e6fb813f, NAME => 'TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp 2023-03-20 00:57:18,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-20 00:57:18,901 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 00:57:18,901 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 718cd5d8d8fa4a8a2d92da64e6fb813f, disabling compactions & flushes 2023-03-20 00:57:18,901 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:18,902 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:18,902 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. after waiting 0 ms 2023-03-20 00:57:18,902 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:18,902 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:18,902 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:18,904 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-20 00:57:18,905 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273838904"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679273838904"}]},"ts":"1679273838904"} 2023-03-20 00:57:18,905 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-20 00:57:18,906 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-20 00:57:18,906 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679273838906"}]},"ts":"1679273838906"} 2023-03-20 00:57:18,907 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-20 00:57:18,909 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=718cd5d8d8fa4a8a2d92da64e6fb813f, ASSIGN}] 2023-03-20 00:57:18,910 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=718cd5d8d8fa4a8a2d92da64e6fb813f, ASSIGN 2023-03-20 00:57:18,911 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=718cd5d8d8fa4a8a2d92da64e6fb813f, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34553,1679273725199; forceNewPlan=false, retain=false 2023-03-20 00:57:19,062 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=718cd5d8d8fa4a8a2d92da64e6fb813f, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:19,063 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273839062"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679273839062"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679273839062"}]},"ts":"1679273839062"} 2023-03-20 00:57:19,064 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199}] 2023-03-20 00:57:19,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-20 00:57:19,219 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:19,220 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 718cd5d8d8fa4a8a2d92da64e6fb813f, NAME => 'TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.', STARTKEY => '', ENDKEY => ''} 2023-03-20 00:57:19,220 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:19,220 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 00:57:19,220 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:19,220 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:19,221 INFO [StoreOpener-718cd5d8d8fa4a8a2d92da64e6fb813f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:19,222 DEBUG [StoreOpener-718cd5d8d8fa4a8a2d92da64e6fb813f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A 2023-03-20 00:57:19,222 DEBUG [StoreOpener-718cd5d8d8fa4a8a2d92da64e6fb813f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A 2023-03-20 00:57:19,222 INFO [StoreOpener-718cd5d8d8fa4a8a2d92da64e6fb813f-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 00:57:19,222 INFO [StoreOpener-718cd5d8d8fa4a8a2d92da64e6fb813f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 718cd5d8d8fa4a8a2d92da64e6fb813f columnFamilyName A 2023-03-20 00:57:19,223 INFO [StoreOpener-718cd5d8d8fa4a8a2d92da64e6fb813f-1] regionserver.HStore(310): Store=718cd5d8d8fa4a8a2d92da64e6fb813f/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:57:19,223 INFO [StoreOpener-718cd5d8d8fa4a8a2d92da64e6fb813f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:19,224 DEBUG [StoreOpener-718cd5d8d8fa4a8a2d92da64e6fb813f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B 2023-03-20 00:57:19,224 DEBUG [StoreOpener-718cd5d8d8fa4a8a2d92da64e6fb813f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B 2023-03-20 00:57:19,225 INFO [StoreOpener-718cd5d8d8fa4a8a2d92da64e6fb813f-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 00:57:19,225 INFO [StoreOpener-718cd5d8d8fa4a8a2d92da64e6fb813f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 718cd5d8d8fa4a8a2d92da64e6fb813f columnFamilyName B 2023-03-20 00:57:19,225 INFO [StoreOpener-718cd5d8d8fa4a8a2d92da64e6fb813f-1] regionserver.HStore(310): Store=718cd5d8d8fa4a8a2d92da64e6fb813f/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:57:19,225 INFO [StoreOpener-718cd5d8d8fa4a8a2d92da64e6fb813f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:19,226 DEBUG [StoreOpener-718cd5d8d8fa4a8a2d92da64e6fb813f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C 2023-03-20 00:57:19,226 DEBUG [StoreOpener-718cd5d8d8fa4a8a2d92da64e6fb813f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C 2023-03-20 00:57:19,227 INFO [StoreOpener-718cd5d8d8fa4a8a2d92da64e6fb813f-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 00:57:19,227 INFO [StoreOpener-718cd5d8d8fa4a8a2d92da64e6fb813f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 718cd5d8d8fa4a8a2d92da64e6fb813f columnFamilyName C 2023-03-20 00:57:19,227 INFO [StoreOpener-718cd5d8d8fa4a8a2d92da64e6fb813f-1] regionserver.HStore(310): Store=718cd5d8d8fa4a8a2d92da64e6fb813f/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:57:19,227 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:19,228 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:19,228 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:19,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-20 00:57:19,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:19,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-20 00:57:19,232 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 718cd5d8d8fa4a8a2d92da64e6fb813f; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69604518, jitterRate=0.03718814253807068}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-20 00:57:19,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:19,233 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f., pid=49, masterSystemTime=1679273839216 2023-03-20 00:57:19,234 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:19,234 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:19,234 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=718cd5d8d8fa4a8a2d92da64e6fb813f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:19,234 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273839234"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679273839234"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679273839234"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679273839234"}]},"ts":"1679273839234"} 2023-03-20 00:57:19,236 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-03-20 00:57:19,236 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 in 171 msec 2023-03-20 00:57:19,237 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-03-20 00:57:19,237 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=718cd5d8d8fa4a8a2d92da64e6fb813f, ASSIGN in 327 msec 2023-03-20 00:57:19,237 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-20 00:57:19,237 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679273839237"}]},"ts":"1679273839237"} 2023-03-20 00:57:19,238 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-20 00:57:19,240 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-20 00:57:19,241 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1600 sec 2023-03-20 00:57:20,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-20 00:57:20,190 INFO [Listener at localhost.localdomain/43361] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-03-20 00:57:20,191 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x51ff7cd4 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:57:20,197 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2e8c6f6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:57:20,198 DEBUG [Listener at localhost.localdomain/43361] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:57:20,200 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52274, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:57:20,201 DEBUG [Listener at localhost.localdomain/43361] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-20 00:57:20,204 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43620, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-20 00:57:20,205 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x682c0b9e to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:57:20,211 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@647f95f6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:57:20,212 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x50d6899d to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:57:20,216 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@58ae18e2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:57:20,216 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x6662e946 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:57:20,220 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29c1e76, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:57:20,221 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x5d7deb76 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:57:20,234 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c240146, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:57:20,234 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x0f4b7807 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:57:20,240 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d5f04c3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:57:20,240 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x6523d347 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:57:20,250 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@209faa3c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:57:20,251 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x28072326 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:57:20,259 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5345748a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:57:20,259 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x2d90797d to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:57:20,265 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c9f61fa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:57:20,265 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x7d3df389 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:57:20,276 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@534826c2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:57:20,276 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x65f69c3f to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:57:20,284 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@77459ab3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:57:20,290 DEBUG [hconnection-0x1e590adf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:57:20,290 DEBUG [hconnection-0x7b46773d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:57:20,290 DEBUG [hconnection-0x406eb0ec-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:57:20,290 DEBUG [hconnection-0x26c68fae-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:57:20,290 DEBUG [hconnection-0x6c499665-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:57:20,291 DEBUG [hconnection-0x19235d32-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:57:20,292 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52294, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:57:20,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:20,292 DEBUG [hconnection-0x130eb58c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:57:20,292 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52320, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:57:20,295 DEBUG [hconnection-0x45267bc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:57:20,295 DEBUG [hconnection-0x4a3ed08-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:57:20,299 DEBUG [hconnection-0x35805782-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:57:20,299 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52280, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:57:20,299 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52300, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:57:20,300 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52302, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:57:20,300 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52332, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:57:20,301 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52350, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:57:20,301 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52338, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:57:20,305 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52358, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:57:20,305 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52362, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:57:20,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:20,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:20,307 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:20,307 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:20,307 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:20,307 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:20,308 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,308 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:20,309 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:20,310 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:20,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:20,311 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,311 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:20,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:20,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:20,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:20,311 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:20,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:20,315 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:20,316 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:20,316 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:20,316 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:20,316 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:20,316 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:20,317 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:20,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:20,317 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:20,317 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 718cd5d8d8fa4a8a2d92da64e6fb813f 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-20 00:57:20,317 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:20,317 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:20,318 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:20,318 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:20,318 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,319 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:20,319 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:20,320 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=A 2023-03-20 00:57:20,320 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:20,320 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=B 2023-03-20 00:57:20,320 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:20,320 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=C 2023-03-20 00:57:20,320 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:20,325 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,325 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:20,325 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,325 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:20,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:20,326 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:20,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d0b2366[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:20,326 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,327 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,327 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:20,327 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:20,327 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:20,327 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,327 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:20,328 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:20,328 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:20,328 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:20,328 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:20,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:20,330 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:20,330 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:20,330 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:20,330 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:20,330 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:20,331 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,331 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:20,331 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:20,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:20,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:20,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:20,332 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:20,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:20,332 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:20,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,335 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:20,335 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,335 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:20,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:20,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:20,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:20,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:20,335 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:20,335 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:20,335 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:20,335 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:20,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:20,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:20,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:20,335 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:20,335 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,336 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,336 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,336 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:20,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273900336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:20,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:20,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273900356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273900356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:20,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:20,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273900356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273900357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,436 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:20,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:20,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:20,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:20,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:20,440 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:20,440 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:20,441 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:20,441 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:20,441 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,441 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:20,441 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:20,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:20,442 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,442 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:20,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:20,442 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:20,446 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:20,446 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:20,446 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:20,446 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:20,446 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:20,448 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:20,448 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:20,448 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:20,448 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:20,448 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:20,448 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:20,448 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:20,448 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:20,448 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,448 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:20,448 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:20,462 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:20,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273900460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,466 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,466 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:20,467 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,467 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:20,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:20,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273900464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:20,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273900464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:20,468 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:20,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273900464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,468 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:20,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273900464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:20,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:20,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15132c8a[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:20,469 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:20,469 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,472 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,473 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:20,473 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:20,473 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:20,473 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,473 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:20,473 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,473 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:20,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:20,473 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:20,474 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:20,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:20,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:20,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:20,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:20,477 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:20,477 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:20,477 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:20,477 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:20,477 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:20,478 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,478 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,478 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:20,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:20,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:20,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,479 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:20,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,480 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,481 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:20,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:20,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:20,481 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,481 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:20,481 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:20,481 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,481 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,481 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,481 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,482 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:20,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:20,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:20,482 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:20,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:20,482 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:20,482 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:20,582 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:20,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:20,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:20,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:20,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:20,585 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:20,585 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:20,585 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:20,585 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:20,585 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,585 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:20,586 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:20,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:20,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:20,588 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:20,588 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,588 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:20,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:20,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:20,588 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:20,589 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:20,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:20,589 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:20,589 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:20,589 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:20,590 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:20,590 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:20,590 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:20,590 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:20,590 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:20,590 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:20,590 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:20,590 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:20,590 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,590 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:20,590 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:20,591 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,591 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:20,591 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:20,591 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:20,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:20,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:20,592 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:20,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40943304[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:20,592 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,593 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,593 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:20,593 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,593 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:20,593 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:20,593 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:20,594 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:20,594 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:20,594 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:20,594 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:20,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:20,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:20,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:20,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:20,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:20,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:20,596 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:20,596 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:20,596 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:20,596 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:20,596 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:20,597 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,597 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:20,597 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:20,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:20,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,597 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:20,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:20,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:20,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:20,598 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:20,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,600 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,600 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:20,600 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:20,600 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:20,600 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:20,600 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,600 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,600 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,600 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:20,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:20,600 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:20,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:20,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:20,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:20,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:20,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:20,600 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:20,600 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:20,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:20,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273900663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:20,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273900668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:20,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273900668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:20,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273900668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:20,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273900668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,700 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:20,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:20,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:20,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:20,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:20,703 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:20,703 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:20,703 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:20,703 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:20,703 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,703 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:20,704 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:20,704 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:20,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:20,704 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,704 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:20,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:20,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:20,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:20,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:20,705 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:20,705 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:20,705 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:20,705 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:20,705 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:20,705 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:20,705 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:20,705 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:20,705 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:20,706 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:20,706 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:20,706 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:20,706 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:20,706 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,706 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:20,706 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:20,706 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,707 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,707 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:20,707 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:20,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:20,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17e7b370[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:20,708 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:20,708 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,708 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,709 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:20,709 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,709 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:20,709 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:20,709 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:20,709 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,709 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:20,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:20,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:20,709 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:20,710 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:20,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:20,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:20,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:20,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:20,711 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:20,711 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:20,711 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:20,711 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:20,711 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:20,712 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,712 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,712 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:20,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:20,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:20,712 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:20,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:20,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,713 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:20,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,714 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,714 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:20,715 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:20,715 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:20,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:20,715 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,715 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:20,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:20,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:20,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:20,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:20,715 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:20,715 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,715 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,715 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,715 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:20,715 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:20,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:20,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:20,739 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/f43d1b30e12c44c0a0814573d9a49e0e 2023-03-20 00:57:20,815 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:20,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:20,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:20,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:20,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:20,818 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:20,818 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:20,818 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:20,818 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:20,818 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,818 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:20,820 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:20,820 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:20,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:20,820 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,820 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:20,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:20,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:20,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:20,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:20,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:20,821 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:20,821 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:20,821 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:20,821 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:20,821 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:20,821 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:20,821 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:20,821 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:20,821 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:20,821 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:20,822 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:20,822 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:20,822 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,822 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:20,822 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:20,822 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,822 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:20,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:20,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:20,823 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,823 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:20,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:20,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:20,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:20,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23c3a2f9[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:20,824 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:20,824 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,824 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,824 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:20,824 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:20,824 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,824 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:20,824 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:20,825 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,825 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:20,825 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:20,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:20,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:20,825 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:20,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:20,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:20,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:20,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:20,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:20,827 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:20,827 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:20,827 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:20,827 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:20,827 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:20,828 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,828 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:20,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:20,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,828 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:20,828 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:20,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:20,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:20,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:20,829 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:20,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,831 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,831 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:20,831 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:20,831 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:20,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:20,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:20,831 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:20,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:20,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:20,831 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:20,831 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,831 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:20,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:20,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:20,832 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,832 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:20,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:20,832 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,832 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,932 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:20,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:20,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:20,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:20,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:20,935 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:20,935 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:20,935 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:20,935 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:20,935 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,935 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:20,936 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,936 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:20,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:20,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:20,936 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,936 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:20,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:20,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:20,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:20,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:20,937 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:20,937 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:20,937 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:20,937 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:20,937 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:20,937 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:20,937 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:20,937 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:20,937 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:20,937 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:20,937 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:20,937 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:20,937 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:20,937 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,938 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:20,938 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:20,938 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,938 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:20,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:20,939 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,939 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:20,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:20,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:20,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:20,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:20,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f38bd77[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:20,940 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:20,940 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,940 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,940 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:20,940 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:20,940 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,940 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:20,940 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:20,941 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:20,941 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:20,941 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,941 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:20,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:20,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:20,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:20,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:20,943 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:20,943 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:20,943 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:20,943 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:20,943 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:20,944 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:20,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:20,945 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:20,945 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,945 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:20,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:20,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:20,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:20,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,946 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:20,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,947 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:20,947 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,947 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:20,947 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:20,947 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:20,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:20,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:20,947 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:20,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:20,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:20,947 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,947 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,947 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:20,947 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:20,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:20,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:20,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:20,948 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:20,948 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:20,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:20,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273900966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:20,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273900972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:20,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273900972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:20,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273900973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:20,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:20,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273900973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,048 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:21,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:21,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:21,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:21,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:21,050 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:21,050 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:21,050 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:21,050 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:21,051 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,051 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:21,051 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:21,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:21,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:21,051 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:21,052 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,052 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:21,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:21,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:21,052 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:21,053 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:21,053 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:21,053 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:21,053 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:21,053 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:21,053 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:21,053 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:21,053 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:21,053 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:21,054 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:21,054 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:21,054 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:21,054 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:21,054 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,054 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:21,054 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:21,055 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,055 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:21,055 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,055 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:21,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:21,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:21,056 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:21,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ed5a787[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:21,056 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,057 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,057 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,057 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:21,057 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:21,057 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:21,057 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:21,058 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,058 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:21,058 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:21,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:21,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:21,058 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:21,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:21,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:21,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:21,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:21,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:21,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:21,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:21,060 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:21,060 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:21,060 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:21,060 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:21,060 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:21,061 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,061 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:21,061 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:21,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:21,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,061 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:21,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:21,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:21,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:21,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:21,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:21,062 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:21,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:21,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,063 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,063 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:21,063 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:21,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:21,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:21,063 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:21,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:21,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:21,063 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:21,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:21,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:21,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:21,064 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:21,064 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:21,064 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:21,064 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,064 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,064 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,064 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,148 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/bff40d9835464a799a6eafb15b86584a 2023-03-20 00:57:21,164 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:21,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:21,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:21,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:21,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:21,166 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:21,167 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:21,167 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:21,167 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:21,167 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,167 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:21,167 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,168 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:21,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:21,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:21,168 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,168 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:21,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:21,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:21,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:21,168 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:21,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:21,168 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:21,168 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:21,168 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:21,169 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:21,169 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:21,169 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:21,169 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:21,169 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:21,169 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:21,169 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:21,169 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:21,169 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,169 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:21,169 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:21,170 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,170 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:21,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:21,170 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,170 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:21,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:21,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:21,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:21,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:21,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:21,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6403f11e[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:21,171 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:21,171 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,172 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,173 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:21,173 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:21,173 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:21,173 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,173 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:21,173 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:21,174 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:21,174 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:21,174 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:21,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:21,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:21,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:21,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:21,175 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:21,175 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:21,175 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:21,175 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:21,175 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:21,176 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:21,176 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,176 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:21,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:21,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:21,177 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,178 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,179 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,179 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:21,179 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:21,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:21,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:21,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:21,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:21,180 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:21,179 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:21,179 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:21,180 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:21,180 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,180 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:21,180 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,180 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,180 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:21,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:21,280 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:21,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:21,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:21,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:21,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:21,283 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:21,283 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:21,283 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:21,283 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:21,283 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,283 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:21,284 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:21,284 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:21,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:21,284 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,284 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:21,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:21,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:21,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:21,285 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:21,285 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:21,285 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:21,285 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:21,285 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:21,285 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:21,286 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:21,286 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:21,286 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:21,286 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:21,286 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:21,286 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:21,286 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:21,286 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:21,286 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,286 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:21,286 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:21,287 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,287 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:21,287 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,287 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:21,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:21,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a69430d[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:21,289 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:21,289 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,289 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,290 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:21,290 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:21,290 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:21,290 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,290 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:21,291 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:21,291 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:21,292 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:21,292 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:21,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:21,293 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:21,293 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:21,293 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:21,293 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:21,293 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:21,294 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,294 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,294 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:21,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:21,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:21,295 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:21,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:21,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:21,296 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:21,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:21,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,298 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:21,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:21,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:21,299 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:21,299 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:21,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:21,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:21,299 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:21,299 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:21,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:21,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:21,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:21,302 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,302 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:21,302 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:21,302 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,302 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,302 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,302 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,399 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:21,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:21,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:21,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:21,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:21,406 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:21,406 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:21,406 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:21,406 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:21,406 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,407 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:21,407 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,407 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:21,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:21,407 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,407 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:21,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:21,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:21,408 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:21,414 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:21,414 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:21,414 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:21,414 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:21,414 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:21,422 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:21,422 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:21,422 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:21,422 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:21,422 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:21,422 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:21,422 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:21,422 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:21,422 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,422 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:21,422 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:21,433 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,433 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:21,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:21,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:21,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:21,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:21,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:21,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:21,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:21,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30508e32[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:21,436 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:21,436 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,437 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,437 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:21,437 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:21,437 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:21,437 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,437 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:21,438 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,439 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:21,439 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:21,439 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:21,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:21,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:21,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:21,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:21,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:21,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:21,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:21,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:21,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:21,441 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:21,441 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:21,441 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:21,441 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:21,441 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:21,442 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:21,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:21,443 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,443 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:21,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:21,443 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:21,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:21,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:21,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:21,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:21,444 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,446 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:21,446 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:21,446 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:21,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:21,446 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:21,446 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,446 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,446 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,446 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,446 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:21,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:21,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:21,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:21,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:21,447 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:21,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:21,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:21,447 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:21,447 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:21,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:21,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273901473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:21,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273901475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,479 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:21,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273901477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,479 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:21,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273901477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:21,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273901479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,547 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:21,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:21,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:21,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:21,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:21,552 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:21,552 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:21,552 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:21,552 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:21,552 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,552 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:21,553 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,553 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:21,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:21,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:21,553 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,553 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:21,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:21,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:21,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:21,554 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:21,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:21,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:21,554 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:21,554 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:21,554 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:21,554 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:21,554 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:21,554 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:21,554 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:21,555 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:21,555 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:21,555 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:21,555 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:21,555 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,555 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:21,555 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:21,556 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/cfa96489f21f4296801226298e8d1b9c 2023-03-20 00:57:21,556 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:21,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:21,556 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:21,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:21,556 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,556 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:21,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:21,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:21,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:21,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53dafe20[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:21,558 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:21,558 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,558 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,559 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:21,559 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:21,559 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:21,560 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,560 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:21,561 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/f43d1b30e12c44c0a0814573d9a49e0e as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/f43d1b30e12c44c0a0814573d9a49e0e 2023-03-20 00:57:21,562 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:21,562 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,562 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:21,562 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:21,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:21,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:21,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:21,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:21,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:21,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:21,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:21,564 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/f43d1b30e12c44c0a0814573d9a49e0e, entries=150, sequenceid=13, filesize=11.6 K 2023-03-20 00:57:21,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,565 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/bff40d9835464a799a6eafb15b86584a as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/bff40d9835464a799a6eafb15b86584a 2023-03-20 00:57:21,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:21,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:21,566 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:21,566 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:21,566 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:21,566 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:21,566 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:21,567 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:21,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:21,567 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,567 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:21,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:21,568 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:21,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:21,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:21,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:21,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:21,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:21,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:21,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,571 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,571 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/bff40d9835464a799a6eafb15b86584a, entries=150, sequenceid=13, filesize=11.6 K 2023-03-20 00:57:21,572 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/cfa96489f21f4296801226298e8d1b9c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/cfa96489f21f4296801226298e8d1b9c 2023-03-20 00:57:21,572 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:21,572 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:21,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:21,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:21,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:21,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:21,573 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,573 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:21,573 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:21,573 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:21,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:21,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:21,573 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,573 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,573 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:21,573 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,574 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:21,574 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:21,577 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/cfa96489f21f4296801226298e8d1b9c, entries=150, sequenceid=13, filesize=11.6 K 2023-03-20 00:57:21,577 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 718cd5d8d8fa4a8a2d92da64e6fb813f in 1260ms, sequenceid=13, compaction requested=false 2023-03-20 00:57:21,577 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:21,674 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:21,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:21,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:21,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:21,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:21,677 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:21,677 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:21,677 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:21,677 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:21,677 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,677 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:21,678 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,678 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:21,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:21,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:21,678 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:21,678 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:21,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:21,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:21,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:21,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:21,679 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:21,679 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:21,679 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:21,679 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:21,679 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:21,679 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:21,682 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:21,682 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:21,682 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:21,682 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:21,682 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 718cd5d8d8fa4a8a2d92da64e6fb813f 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-20 00:57:21,682 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=A 2023-03-20 00:57:21,682 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:21,682 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=B 2023-03-20 00:57:21,682 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:21,682 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=C 2023-03-20 00:57:21,682 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:22,093 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/d939a57e3ba14e1abbe64457effe95fc 2023-03-20 00:57:22,303 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-20 00:57:22,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:22,485 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:22,509 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/b7114eba9aa447179319b4bb94d5ab1f 2023-03-20 00:57:22,521 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/a5d74c88ab6b4969a382bfb0bb0ea15b 2023-03-20 00:57:22,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:22,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273902515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:22,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273902517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:22,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273902518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,524 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/d939a57e3ba14e1abbe64457effe95fc as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/d939a57e3ba14e1abbe64457effe95fc 2023-03-20 00:57:22,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:22,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273902521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,528 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/d939a57e3ba14e1abbe64457effe95fc, entries=150, sequenceid=33, filesize=11.6 K 2023-03-20 00:57:22,528 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/b7114eba9aa447179319b4bb94d5ab1f as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/b7114eba9aa447179319b4bb94d5ab1f 2023-03-20 00:57:22,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:22,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273902524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,538 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/b7114eba9aa447179319b4bb94d5ab1f, entries=150, sequenceid=33, filesize=11.6 K 2023-03-20 00:57:22,540 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/a5d74c88ab6b4969a382bfb0bb0ea15b as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/a5d74c88ab6b4969a382bfb0bb0ea15b 2023-03-20 00:57:22,543 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/a5d74c88ab6b4969a382bfb0bb0ea15b, entries=150, sequenceid=33, filesize=11.6 K 2023-03-20 00:57:22,547 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 718cd5d8d8fa4a8a2d92da64e6fb813f in 865ms, sequenceid=33, compaction requested=false 2023-03-20 00:57:22,552 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:22,552 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:22,553 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:22,553 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:22,553 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,553 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:22,553 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:22,554 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,554 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:22,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:22,554 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,554 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:22,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:22,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:22,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:22,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:22,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:22,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@282a7e44[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:22,557 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:22,557 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,570 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,570 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:22,570 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:22,570 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:22,575 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:22,575 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:22,575 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,575 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:22,575 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:22,575 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:22,575 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:22,575 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:22,575 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:22,575 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:22,575 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:22,581 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:22,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:22,581 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,581 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:22,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:22,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:22,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:22,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:22,582 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:22,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:22,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:22,583 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:22,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,591 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,591 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:22,591 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:22,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:22,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:22,592 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,591 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:22,591 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:22,592 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:22,592 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,592 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:22,592 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:22,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:22,592 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:22,592 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:22,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:22,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:22,628 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 718cd5d8d8fa4a8a2d92da64e6fb813f 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-20 00:57:22,629 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=A 2023-03-20 00:57:22,629 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:22,629 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=B 2023-03-20 00:57:22,629 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:22,629 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=C 2023-03-20 00:57:22,629 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:22,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:22,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273902663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:22,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273902663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:22,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273902663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:22,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273902664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:22,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273902664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,692 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:22,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:22,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:22,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:22,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:22,695 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:22,695 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:22,695 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:22,695 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:22,695 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,695 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:22,696 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,696 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:22,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:22,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:22,696 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,696 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:22,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:22,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:22,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:22,697 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:22,697 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:22,697 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:22,697 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:22,697 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:22,697 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:22,697 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:22,697 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:22,697 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:22,697 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:22,697 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:22,697 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:22,697 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:22,697 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:22,697 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,697 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:22,698 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:22,698 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,698 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:22,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:22,699 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:22,699 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:22,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:22,701 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:22,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38cb2167[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:22,701 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,702 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,702 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:22,702 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,702 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:22,702 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:22,702 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:22,703 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:22,703 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,703 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:22,703 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:22,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:22,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:22,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:22,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:22,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:22,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:22,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:22,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:22,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:22,706 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:22,706 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:22,706 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:22,706 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:22,707 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:22,707 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:22,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:22,707 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,707 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:22,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:22,708 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:22,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:22,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:22,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:22,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:22,710 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:22,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:22,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,711 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,711 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:22,711 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:22,711 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,711 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:22,711 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:22,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:22,711 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,711 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:22,711 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:22,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:22,711 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:22,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:22,711 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:22,712 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:22,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:22,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:22,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:22,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273902769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:22,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273902769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:22,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:22,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273902769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273902770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:22,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273902770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,812 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:22,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:22,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:22,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:22,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:22,814 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:22,814 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:22,814 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:22,814 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:22,815 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,815 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:22,815 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:22,815 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:22,815 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,815 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:22,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:22,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:22,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:22,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:22,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:22,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:22,816 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:22,816 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:22,816 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:22,816 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:22,816 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:22,816 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:22,817 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:22,817 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:22,817 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:22,817 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:22,817 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:22,817 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,817 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:22,817 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:22,818 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,818 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:22,818 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,818 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:22,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:22,819 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:22,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c5e0952[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:22,819 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,820 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,820 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:22,820 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,820 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:22,820 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:22,820 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:22,821 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,821 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:22,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:22,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:22,821 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:22,821 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:22,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:22,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:22,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:22,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:22,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:22,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:22,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:22,823 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:22,823 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:22,823 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:22,823 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:22,823 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:22,823 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,823 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:22,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,824 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:22,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:22,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:22,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:22,824 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:22,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:22,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:22,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:22,825 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,827 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:22,827 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:22,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:22,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:22,827 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:22,827 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:22,827 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:22,828 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:22,827 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:22,828 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:22,828 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:22,828 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,828 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,828 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:22,928 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:22,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:22,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:22,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:22,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:22,932 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:22,932 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:22,932 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:22,932 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:22,932 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,932 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:22,933 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,933 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:22,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:22,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:22,933 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,933 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:22,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:22,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:22,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:22,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:22,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:22,934 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:22,934 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:22,934 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:22,934 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:22,934 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:22,934 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:22,935 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:22,935 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:22,935 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:22,935 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:22,935 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:22,935 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,935 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:22,935 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:22,936 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,936 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,936 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:22,936 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:22,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:22,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:22,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:22,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:22,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:22,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:22,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8f1dd07[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:22,938 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:22,938 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,939 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,939 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:22,939 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:22,939 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:22,939 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,939 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:22,940 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:22,940 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:22,940 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:22,940 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:22,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:22,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:22,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:22,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:22,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:22,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:22,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:22,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:22,944 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:22,944 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:22,944 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:22,944 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:22,944 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:22,945 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,945 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:22,945 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:22,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:22,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:22,947 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:22,949 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:22,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,950 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:22,950 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:22,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:22,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:22,950 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:22,950 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:22,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:22,950 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:22,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:22,950 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:22,951 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:22,951 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:22,951 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,951 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,951 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:22,951 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:22,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:22,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273902975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:22,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273902975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:22,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273902975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:22,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273902977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:22,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:22,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273902978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,036 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/837944bcf4ad45c3910f1105dc91972c 2023-03-20 00:57:23,044 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/a2daf73da87a4fd19d1d2912db845805 2023-03-20 00:57:23,051 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:23,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:23,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:23,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:23,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:23,054 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:23,054 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:23,054 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:23,054 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:23,056 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,056 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:23,056 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:23,056 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:23,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:23,056 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,056 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:23,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:23,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:23,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:23,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:23,057 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:23,057 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:23,057 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:23,057 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:23,057 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:23,057 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:23,057 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:23,057 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:23,057 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:23,057 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:23,057 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:23,057 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:23,058 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:23,058 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,058 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:23,058 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:23,058 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,058 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:23,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:23,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:23,059 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,059 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:23,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:23,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:23,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:23,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:23,060 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:23,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73524a03[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:23,060 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,060 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,060 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:23,060 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,060 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:23,060 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:23,060 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:23,061 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,061 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:23,061 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:23,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:23,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:23,061 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:23,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:23,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:23,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:23,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:23,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:23,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:23,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:23,063 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:23,063 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:23,063 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:23,063 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:23,063 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:23,064 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:23,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:23,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:23,073 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,073 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:23,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,073 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:23,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:23,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:23,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:23,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:23,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,074 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:23,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:23,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,075 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,075 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:23,075 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:23,075 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:23,075 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:23,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:23,075 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,075 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,075 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,076 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:23,076 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:23,076 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:23,076 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:23,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:23,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:23,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:23,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:23,176 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:23,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:23,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:23,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:23,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:23,182 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:23,182 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:23,182 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:23,182 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:23,182 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,182 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:23,183 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:23,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:23,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:23,183 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:23,184 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,184 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:23,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:23,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:23,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:23,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:23,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:23,184 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:23,184 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:23,184 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:23,185 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:23,185 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:23,185 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:23,185 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:23,185 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:23,185 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:23,185 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:23,185 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:23,185 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,185 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:23,185 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:23,186 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,186 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,186 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:23,186 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:23,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:23,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:23,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:23,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:23,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:23,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:23,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12738d17[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:23,188 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:23,188 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,189 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,189 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:23,189 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,189 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:23,189 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:23,189 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:23,190 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,190 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:23,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:23,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:23,190 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:23,191 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:23,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:23,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:23,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:23,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:23,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:23,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:23,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:23,192 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:23,192 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:23,192 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:23,192 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:23,192 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:23,193 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,193 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,193 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:23,194 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:23,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:23,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:23,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:23,196 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:23,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:23,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,199 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:23,199 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:23,199 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:23,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:23,199 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:23,199 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:23,199 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,199 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,199 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,199 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,199 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:23,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:23,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:23,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:23,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:23,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:23,199 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:23,199 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:23,277 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:23,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273903277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,277 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:23,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273903277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:23,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273903277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,286 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:23,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273903281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,286 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:23,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273903282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,299 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:23,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:23,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:23,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:23,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:23,303 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:23,303 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:23,303 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:23,303 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:23,303 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,303 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:23,304 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:23,304 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:23,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:23,304 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,304 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:23,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:23,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:23,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:23,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:23,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:23,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:23,305 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:23,305 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:23,305 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:23,305 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:23,305 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:23,305 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:23,305 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:23,305 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:23,305 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:23,305 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:23,305 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:23,305 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,305 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:23,305 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:23,306 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,306 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:23,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:23,306 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,306 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:23,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:23,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:23,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:23,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:23,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:23,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e9b0292[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:23,308 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:23,308 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,308 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,309 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:23,309 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:23,309 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:23,309 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,309 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:23,309 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:23,309 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:23,309 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:23,310 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:23,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:23,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:23,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:23,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:23,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:23,313 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:23,313 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:23,313 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:23,313 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:23,313 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:23,314 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,314 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:23,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:23,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,314 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:23,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:23,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:23,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:23,315 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:23,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:23,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:23,317 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:23,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:23,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,325 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,325 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:23,325 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:23,325 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:23,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:23,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:23,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:23,325 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:23,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:23,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:23,325 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:23,325 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,325 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,325 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:23,325 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:23,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:23,325 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,325 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:23,426 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:23,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:23,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:23,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:23,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:23,428 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:23,428 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:23,428 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:23,428 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:23,429 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,429 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:23,429 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,429 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:23,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:23,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:23,430 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,430 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:23,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:23,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:23,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:23,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:23,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:23,431 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:23,431 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:23,431 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:23,431 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:23,431 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:23,431 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:23,431 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:23,431 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:23,431 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:23,431 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:23,431 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:23,431 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,431 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:23,431 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:23,432 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,432 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:23,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:23,432 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,432 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:23,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:23,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:23,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:23,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:23,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:23,433 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:23,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6884e3a3[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:23,433 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,434 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:23,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:23,434 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,434 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:23,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:23,435 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,435 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:23,435 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:23,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:23,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:23,435 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:23,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:23,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:23,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:23,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:23,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:23,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:23,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:23,437 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:23,437 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:23,437 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:23,437 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:23,437 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:23,438 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,438 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:23,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,438 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:23,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:23,439 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:23,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:23,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:23,439 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:23,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:23,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:23,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:23,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:23,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:23,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:23,441 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:23,441 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:23,441 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:23,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:23,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:23,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:23,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,456 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/b64dc294974c46e6aeaed195c989b402 2023-03-20 00:57:23,460 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/837944bcf4ad45c3910f1105dc91972c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/837944bcf4ad45c3910f1105dc91972c 2023-03-20 00:57:23,465 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/837944bcf4ad45c3910f1105dc91972c, entries=250, sequenceid=48, filesize=16.2 K 2023-03-20 00:57:23,466 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/a2daf73da87a4fd19d1d2912db845805 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/a2daf73da87a4fd19d1d2912db845805 2023-03-20 00:57:23,472 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/a2daf73da87a4fd19d1d2912db845805, entries=150, sequenceid=48, filesize=11.6 K 2023-03-20 00:57:23,473 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/b64dc294974c46e6aeaed195c989b402 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/b64dc294974c46e6aeaed195c989b402 2023-03-20 00:57:23,477 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/b64dc294974c46e6aeaed195c989b402, entries=150, sequenceid=48, filesize=11.6 K 2023-03-20 00:57:23,478 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 718cd5d8d8fa4a8a2d92da64e6fb813f in 850ms, sequenceid=48, compaction requested=true 2023-03-20 00:57:23,478 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:23,478 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:23,478 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:57:23,478 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:57:23,478 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:23,478 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:23,479 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:57:23,479 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 718cd5d8d8fa4a8a2d92da64e6fb813f/B is initiating minor compaction (all files) 2023-03-20 00:57:23,479 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 718cd5d8d8fa4a8a2d92da64e6fb813f/B in TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:23,479 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/bff40d9835464a799a6eafb15b86584a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/b7114eba9aa447179319b4bb94d5ab1f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/a2daf73da87a4fd19d1d2912db845805] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp, totalSize=34.8 K 2023-03-20 00:57:23,480 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40338 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:57:23,480 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 718cd5d8d8fa4a8a2d92da64e6fb813f/A is initiating minor compaction (all files) 2023-03-20 00:57:23,480 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 718cd5d8d8fa4a8a2d92da64e6fb813f/A in TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:23,480 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/f43d1b30e12c44c0a0814573d9a49e0e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/d939a57e3ba14e1abbe64457effe95fc, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/837944bcf4ad45c3910f1105dc91972c] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp, totalSize=39.4 K 2023-03-20 00:57:23,482 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting bff40d9835464a799a6eafb15b86584a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1679273840310 2023-03-20 00:57:23,482 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting f43d1b30e12c44c0a0814573d9a49e0e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1679273840310 2023-03-20 00:57:23,482 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting b7114eba9aa447179319b4bb94d5ab1f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1679273840333 2023-03-20 00:57:23,483 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting d939a57e3ba14e1abbe64457effe95fc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1679273840333 2023-03-20 00:57:23,483 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 837944bcf4ad45c3910f1105dc91972c, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679273842501 2023-03-20 00:57:23,483 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting a2daf73da87a4fd19d1d2912db845805, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679273842519 2023-03-20 00:57:23,497 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 718cd5d8d8fa4a8a2d92da64e6fb813f#B#compaction#343 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:23,497 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 718cd5d8d8fa4a8a2d92da64e6fb813f#A#compaction#342 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:23,541 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:23,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:23,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:23,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:23,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:23,545 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:23,545 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:23,545 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:23,545 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:23,545 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,545 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:23,546 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:23,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:23,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:23,546 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:23,547 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,547 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:23,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:23,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:23,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:23,548 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:23,550 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:23,550 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:23,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:23,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:23,550 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:23,551 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:23,551 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:23,551 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:23,551 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:23,551 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 718cd5d8d8fa4a8a2d92da64e6fb813f 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-20 00:57:23,551 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=A 2023-03-20 00:57:23,551 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:23,552 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=B 2023-03-20 00:57:23,552 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:23,552 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=C 2023-03-20 00:57:23,552 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:23,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:23,782 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:23,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:23,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273903794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:23,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273903796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,805 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:23,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273903797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,805 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:23,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273903798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,805 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:23,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273903800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,906 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:23,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273903902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,906 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:23,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273903905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,906 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:23,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273903905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,907 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/b3b3403c3f284f4f84180bfc48f5194f as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/b3b3403c3f284f4f84180bfc48f5194f 2023-03-20 00:57:23,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:23,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273903906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:23,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273903906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:23,911 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 718cd5d8d8fa4a8a2d92da64e6fb813f/B of 718cd5d8d8fa4a8a2d92da64e6fb813f into b3b3403c3f284f4f84180bfc48f5194f(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:23,911 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:23,911 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f., storeName=718cd5d8d8fa4a8a2d92da64e6fb813f/B, priority=13, startTime=1679273843478; duration=0sec 2023-03-20 00:57:23,911 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:23,911 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:57:23,912 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:57:23,912 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 718cd5d8d8fa4a8a2d92da64e6fb813f/C is initiating minor compaction (all files) 2023-03-20 00:57:23,912 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 718cd5d8d8fa4a8a2d92da64e6fb813f/C in TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:23,912 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/cfa96489f21f4296801226298e8d1b9c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/a5d74c88ab6b4969a382bfb0bb0ea15b, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/b64dc294974c46e6aeaed195c989b402] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp, totalSize=34.8 K 2023-03-20 00:57:23,912 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting cfa96489f21f4296801226298e8d1b9c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1679273840310 2023-03-20 00:57:23,912 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting a5d74c88ab6b4969a382bfb0bb0ea15b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1679273840333 2023-03-20 00:57:23,913 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting b64dc294974c46e6aeaed195c989b402, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679273842519 2023-03-20 00:57:23,917 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 718cd5d8d8fa4a8a2d92da64e6fb813f#C#compaction#345 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:23,921 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/6018a06a6c2f4165ad95352af606e71c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/6018a06a6c2f4165ad95352af606e71c 2023-03-20 00:57:23,931 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 718cd5d8d8fa4a8a2d92da64e6fb813f/A of 718cd5d8d8fa4a8a2d92da64e6fb813f into 6018a06a6c2f4165ad95352af606e71c(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:23,931 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:23,931 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f., storeName=718cd5d8d8fa4a8a2d92da64e6fb813f/A, priority=13, startTime=1679273843478; duration=0sec 2023-03-20 00:57:23,931 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:23,942 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/132da6828d6f48518df807f41d05b1df as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/132da6828d6f48518df807f41d05b1df 2023-03-20 00:57:23,945 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 718cd5d8d8fa4a8a2d92da64e6fb813f/C of 718cd5d8d8fa4a8a2d92da64e6fb813f into 132da6828d6f48518df807f41d05b1df(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:23,945 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:23,946 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f., storeName=718cd5d8d8fa4a8a2d92da64e6fb813f/C, priority=13, startTime=1679273843478; duration=0sec 2023-03-20 00:57:23,946 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:23,964 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/1eb1df08c6024eedbaeca8b3d7d6d034 2023-03-20 00:57:24,108 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:24,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273904107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:24,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273904108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:24,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273904108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:24,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273904110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:24,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273904110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,375 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/2508c9659d304d3287e838f7975d4e3e 2023-03-20 00:57:24,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:24,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273904410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:24,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273904411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:24,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273904411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:24,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273904413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,419 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:24,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273904415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,786 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/06b8272673494d96a6b9f6ce13dc79d9 2023-03-20 00:57:24,791 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/1eb1df08c6024eedbaeca8b3d7d6d034 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/1eb1df08c6024eedbaeca8b3d7d6d034 2023-03-20 00:57:24,804 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/1eb1df08c6024eedbaeca8b3d7d6d034, entries=150, sequenceid=65, filesize=11.6 K 2023-03-20 00:57:24,805 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/2508c9659d304d3287e838f7975d4e3e as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/2508c9659d304d3287e838f7975d4e3e 2023-03-20 00:57:24,826 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/2508c9659d304d3287e838f7975d4e3e, entries=150, sequenceid=65, filesize=11.6 K 2023-03-20 00:57:24,832 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/06b8272673494d96a6b9f6ce13dc79d9 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/06b8272673494d96a6b9f6ce13dc79d9 2023-03-20 00:57:24,851 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/06b8272673494d96a6b9f6ce13dc79d9, entries=150, sequenceid=65, filesize=11.6 K 2023-03-20 00:57:24,852 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 718cd5d8d8fa4a8a2d92da64e6fb813f in 1301ms, sequenceid=65, compaction requested=false 2023-03-20 00:57:24,852 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:24,852 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:24,852 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:24,852 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:24,852 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,852 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:24,852 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:24,853 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:24,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:24,853 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:24,854 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:24,854 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:24,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:24,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9e7aee2[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:24,855 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:24,855 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:24,855 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:24,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:24,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:24,856 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:24,856 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:24,856 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:24,856 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,856 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:24,856 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,856 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:24,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:24,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:24,856 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:24,857 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:24,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:24,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:24,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:24,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:24,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:24,859 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:24,859 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:24,859 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:24,859 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:24,859 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:24,860 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:24,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:24,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:24,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:24,861 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:24,861 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:24,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:24,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:24,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:24,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:24,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:24,865 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:24,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,866 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:24,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:24,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,875 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,875 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:24,875 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:24,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:24,875 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:24,875 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,875 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:24,875 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:24,875 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:24,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:24,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:24,875 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:24,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:24,876 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:24,876 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:24,875 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:24,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:24,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:24,920 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 718cd5d8d8fa4a8a2d92da64e6fb813f 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-20 00:57:24,921 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=A 2023-03-20 00:57:24,921 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:24,921 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=B 2023-03-20 00:57:24,921 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:24,921 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=C 2023-03-20 00:57:24,921 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:24,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:24,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273904929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,940 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/34c1f4ac47fa471e9b3ad08616965735 2023-03-20 00:57:24,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:24,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273904933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,941 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:24,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273904934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:24,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273904940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,948 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:24,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273904940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,949 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/8ffea4882e8242bbb1e414ea261b5707 2023-03-20 00:57:24,976 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:24,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:24,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:24,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:24,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:24,980 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:24,980 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:24,980 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:24,980 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:24,980 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:24,980 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:24,982 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:24,982 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:24,982 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:24,982 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:24,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:24,983 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:24,983 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:24,983 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:24,983 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:24,983 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:24,983 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:24,984 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:24,984 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:24,984 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:24,984 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:24,984 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:24,985 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:24,985 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:24,985 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:24,985 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,985 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:24,985 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:24,986 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,986 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:24,986 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:24,986 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:24,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:24,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77291fc[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:24,987 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:24,987 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:24,988 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:24,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:24,988 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,988 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:24,988 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:24,988 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:24,988 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:24,989 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:24,989 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:24,989 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:24,989 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:24,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:24,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:24,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:24,992 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:24,992 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:24,992 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:24,993 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:24,993 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:24,994 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:24,994 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:24,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:24,994 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:24,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:24,995 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:24,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:24,996 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,998 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:24,998 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:24,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:24,998 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,998 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:24,998 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:24,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:24,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:24,998 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:24,998 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:24,998 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:24,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:24,998 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:24,998 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:24,998 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:24,998 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:25,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:25,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273905041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:25,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273905041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:25,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273905042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,050 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:25,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273905049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,050 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:25,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273905049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,098 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:25,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:25,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:25,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:25,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:25,101 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:25,101 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:25,101 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:25,101 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:25,101 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:25,102 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:25,102 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,102 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:25,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:25,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:25,102 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,102 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:25,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:25,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:25,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:25,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:25,103 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:25,103 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:25,103 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:25,103 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:25,103 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:25,103 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:25,103 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:25,103 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:25,103 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:25,104 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:25,104 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:25,104 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:25,104 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:25,104 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:25,104 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,104 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:25,104 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:25,104 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:25,105 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:25,105 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:25,105 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:25,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:25,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31cb8c12[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:25,106 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:25,106 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:25,106 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:25,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:25,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:25,107 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:25,107 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:25,107 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:25,107 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,107 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:25,107 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,107 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:25,107 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:25,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:25,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:25,107 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:25,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:25,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:25,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:25,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:25,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:25,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:25,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:25,109 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:25,109 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:25,109 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:25,109 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:25,109 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:25,110 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:25,110 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:25,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:25,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:25,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:25,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:25,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:25,110 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:25,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:25,111 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:25,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:25,112 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,113 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,113 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:25,113 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:25,113 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:25,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:25,113 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:25,113 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:25,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:25,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:25,114 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:25,114 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:25,114 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:25,114 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:25,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:25,118 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:25,118 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:25,218 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:25,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:25,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:25,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:25,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:25,221 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:25,221 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:25,221 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:25,221 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:25,221 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:25,221 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:25,222 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,222 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:25,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:25,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:25,222 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,222 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:25,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:25,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:25,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:25,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:25,223 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:25,223 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:25,223 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:25,223 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:25,223 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:25,223 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:25,223 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:25,223 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:25,223 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:25,223 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:25,223 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:25,223 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:25,223 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:25,223 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:25,223 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,224 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:25,224 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:25,224 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,224 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:25,225 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:25,225 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:25,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:25,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:25,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:25,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:25,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:25,226 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:25,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e9de873[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:25,226 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:25,226 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:25,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:25,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:25,226 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:25,226 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:25,226 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:25,226 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,226 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:25,227 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,227 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:25,227 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:25,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:25,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:25,227 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:25,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:25,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:25,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:25,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:25,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:25,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:25,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:25,229 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:25,229 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:25,229 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:25,229 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:25,229 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:25,230 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:25,230 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:25,231 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:25,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:25,231 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:25,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:25,232 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:25,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,233 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,233 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:25,233 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:25,233 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:25,233 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:25,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:25,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:25,233 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,233 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:25,233 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:25,233 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:25,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:25,234 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:25,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:25,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:25,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:25,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:25,234 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:25,234 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:25,247 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:25,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273905246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,248 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:25,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273905246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,248 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:25,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273905246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,252 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:25,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273905250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,252 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:25,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273905251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,334 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:25,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:25,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:25,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:25,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:25,336 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:25,337 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:25,337 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:25,337 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:25,337 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:25,337 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:25,338 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:25,338 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:25,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:25,338 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,338 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:25,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:25,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:25,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:25,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:25,338 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:25,339 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:25,339 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:25,339 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:25,339 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:25,339 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:25,339 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:25,339 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:25,339 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:25,339 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:25,339 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:25,339 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:25,339 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:25,339 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:25,339 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,339 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:25,339 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:25,341 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:25,341 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:25,341 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:25,341 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:25,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:25,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ab9d9fc[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:25,342 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:25,342 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:25,343 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:25,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:25,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:25,343 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:25,343 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,343 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:25,343 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:25,343 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:25,344 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,344 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:25,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:25,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:25,344 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:25,344 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:25,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:25,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:25,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:25,347 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:25,347 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:25,347 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:25,347 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:25,347 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:25,348 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:25,348 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:25,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:25,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:25,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:25,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:25,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:25,348 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:25,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:25,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:25,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:25,348 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:25,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:25,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:25,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:25,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,349 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:25,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:25,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,350 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,350 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:25,350 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:25,350 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,350 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:25,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:25,350 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:25,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:25,350 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:25,351 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:25,350 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:25,351 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:25,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:25,350 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:25,352 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:25,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:25,356 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/3be9559f58c24116b979e74cd14f5474 2023-03-20 00:57:25,359 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/34c1f4ac47fa471e9b3ad08616965735 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/34c1f4ac47fa471e9b3ad08616965735 2023-03-20 00:57:25,361 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/34c1f4ac47fa471e9b3ad08616965735, entries=150, sequenceid=85, filesize=11.6 K 2023-03-20 00:57:25,362 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/8ffea4882e8242bbb1e414ea261b5707 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/8ffea4882e8242bbb1e414ea261b5707 2023-03-20 00:57:25,364 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/8ffea4882e8242bbb1e414ea261b5707, entries=100, sequenceid=85, filesize=9.3 K 2023-03-20 00:57:25,365 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/3be9559f58c24116b979e74cd14f5474 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/3be9559f58c24116b979e74cd14f5474 2023-03-20 00:57:25,375 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/3be9559f58c24116b979e74cd14f5474, entries=100, sequenceid=85, filesize=9.3 K 2023-03-20 00:57:25,376 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 718cd5d8d8fa4a8a2d92da64e6fb813f in 456ms, sequenceid=85, compaction requested=true 2023-03-20 00:57:25,376 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:25,376 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:25,376 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:57:25,376 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:57:25,376 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:25,376 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:25,377 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35757 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:57:25,377 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 718cd5d8d8fa4a8a2d92da64e6fb813f/A is initiating minor compaction (all files) 2023-03-20 00:57:25,377 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33417 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:57:25,377 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 718cd5d8d8fa4a8a2d92da64e6fb813f/A in TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:25,377 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 718cd5d8d8fa4a8a2d92da64e6fb813f/B is initiating minor compaction (all files) 2023-03-20 00:57:25,377 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/6018a06a6c2f4165ad95352af606e71c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/1eb1df08c6024eedbaeca8b3d7d6d034, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/34c1f4ac47fa471e9b3ad08616965735] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp, totalSize=34.9 K 2023-03-20 00:57:25,377 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 718cd5d8d8fa4a8a2d92da64e6fb813f/B in TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:25,377 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/b3b3403c3f284f4f84180bfc48f5194f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/2508c9659d304d3287e838f7975d4e3e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/8ffea4882e8242bbb1e414ea261b5707] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp, totalSize=32.6 K 2023-03-20 00:57:25,380 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 6018a06a6c2f4165ad95352af606e71c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679273842519 2023-03-20 00:57:25,381 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting b3b3403c3f284f4f84180bfc48f5194f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679273842519 2023-03-20 00:57:25,381 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 1eb1df08c6024eedbaeca8b3d7d6d034, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1679273842636 2023-03-20 00:57:25,381 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 2508c9659d304d3287e838f7975d4e3e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1679273842636 2023-03-20 00:57:25,381 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 34c1f4ac47fa471e9b3ad08616965735, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1679273843793 2023-03-20 00:57:25,381 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 8ffea4882e8242bbb1e414ea261b5707, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1679273843793 2023-03-20 00:57:25,387 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 718cd5d8d8fa4a8a2d92da64e6fb813f#A#compaction#352 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:25,388 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 718cd5d8d8fa4a8a2d92da64e6fb813f#B#compaction#351 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:25,414 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/69129982a8c34a629d9eba3c2f114346 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/69129982a8c34a629d9eba3c2f114346 2023-03-20 00:57:25,418 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 718cd5d8d8fa4a8a2d92da64e6fb813f/A of 718cd5d8d8fa4a8a2d92da64e6fb813f into 69129982a8c34a629d9eba3c2f114346(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:25,418 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:25,419 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f., storeName=718cd5d8d8fa4a8a2d92da64e6fb813f/A, priority=13, startTime=1679273845376; duration=0sec 2023-03-20 00:57:25,419 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:25,419 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:57:25,421 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33417 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:57:25,421 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 718cd5d8d8fa4a8a2d92da64e6fb813f/C is initiating minor compaction (all files) 2023-03-20 00:57:25,421 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 718cd5d8d8fa4a8a2d92da64e6fb813f/C in TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:25,421 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/132da6828d6f48518df807f41d05b1df, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/06b8272673494d96a6b9f6ce13dc79d9, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/3be9559f58c24116b979e74cd14f5474] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp, totalSize=32.6 K 2023-03-20 00:57:25,421 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 132da6828d6f48518df807f41d05b1df, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679273842519 2023-03-20 00:57:25,421 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 06b8272673494d96a6b9f6ce13dc79d9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1679273842636 2023-03-20 00:57:25,422 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 3be9559f58c24116b979e74cd14f5474, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1679273843793 2023-03-20 00:57:25,429 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 718cd5d8d8fa4a8a2d92da64e6fb813f#C#compaction#353 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:25,452 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:25,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:25,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:25,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:25,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:25,460 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:25,460 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:25,460 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:25,460 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:25,460 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:25,460 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:25,461 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:25,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:25,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:25,461 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:25,461 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,461 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:25,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:25,462 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:25,465 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:25,465 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:25,465 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:25,465 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:25,465 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:25,465 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:25,465 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:25,465 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:25,465 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:25,465 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 718cd5d8d8fa4a8a2d92da64e6fb813f 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-20 00:57:25,466 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=A 2023-03-20 00:57:25,466 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:25,466 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=B 2023-03-20 00:57:25,466 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:25,466 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=C 2023-03-20 00:57:25,466 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:25,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:25,551 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:25,569 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:25,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273905563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,569 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:25,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273905564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:25,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273905568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:25,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273905569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:25,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273905570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:25,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273905670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:25,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273905670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:25,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273905675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:25,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273905675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:25,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273905675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,803 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/b9cdbc543018454d9a6aa4e70d5b224e as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/b9cdbc543018454d9a6aa4e70d5b224e 2023-03-20 00:57:25,806 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 718cd5d8d8fa4a8a2d92da64e6fb813f/B of 718cd5d8d8fa4a8a2d92da64e6fb813f into b9cdbc543018454d9a6aa4e70d5b224e(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:25,806 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:25,806 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f., storeName=718cd5d8d8fa4a8a2d92da64e6fb813f/B, priority=13, startTime=1679273845376; duration=0sec 2023-03-20 00:57:25,806 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:25,846 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/132109c216d34c63ba19e48566e93b65 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/132109c216d34c63ba19e48566e93b65 2023-03-20 00:57:25,849 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 718cd5d8d8fa4a8a2d92da64e6fb813f/C of 718cd5d8d8fa4a8a2d92da64e6fb813f into 132109c216d34c63ba19e48566e93b65(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:25,849 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:25,849 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f., storeName=718cd5d8d8fa4a8a2d92da64e6fb813f/C, priority=13, startTime=1679273845376; duration=0sec 2023-03-20 00:57:25,849 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:25,874 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/f4ef8b79b1084bd292623ed21d268edc 2023-03-20 00:57:25,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:25,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273905876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:25,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273905877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:25,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273905883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:25,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273905883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:25,888 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:25,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273905884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:26,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273906180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:26,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273906183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,187 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:26,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273906184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:26,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273906186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:26,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273906190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,279 DEBUG [regionserver/jenkins-hbase20:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-20 00:57:26,280 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/f43d1b30e12c44c0a0814573d9a49e0e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/d939a57e3ba14e1abbe64457effe95fc, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/837944bcf4ad45c3910f1105dc91972c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/6018a06a6c2f4165ad95352af606e71c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/1eb1df08c6024eedbaeca8b3d7d6d034, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/34c1f4ac47fa471e9b3ad08616965735] to archive 2023-03-20 00:57:26,281 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 00:57:26,282 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/f43d1b30e12c44c0a0814573d9a49e0e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/f43d1b30e12c44c0a0814573d9a49e0e 2023-03-20 00:57:26,283 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/7f9d98ce5d2943c4a80820d80c66fbe9 2023-03-20 00:57:26,284 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/d939a57e3ba14e1abbe64457effe95fc to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/d939a57e3ba14e1abbe64457effe95fc 2023-03-20 00:57:26,284 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/837944bcf4ad45c3910f1105dc91972c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/837944bcf4ad45c3910f1105dc91972c 2023-03-20 00:57:26,285 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/6018a06a6c2f4165ad95352af606e71c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/6018a06a6c2f4165ad95352af606e71c 2023-03-20 00:57:26,286 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/1eb1df08c6024eedbaeca8b3d7d6d034 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/1eb1df08c6024eedbaeca8b3d7d6d034 2023-03-20 00:57:26,287 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/34c1f4ac47fa471e9b3ad08616965735 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/34c1f4ac47fa471e9b3ad08616965735 2023-03-20 00:57:26,290 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/bff40d9835464a799a6eafb15b86584a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/b7114eba9aa447179319b4bb94d5ab1f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/b3b3403c3f284f4f84180bfc48f5194f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/a2daf73da87a4fd19d1d2912db845805, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/2508c9659d304d3287e838f7975d4e3e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/8ffea4882e8242bbb1e414ea261b5707] to archive 2023-03-20 00:57:26,291 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 00:57:26,294 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/bff40d9835464a799a6eafb15b86584a to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/bff40d9835464a799a6eafb15b86584a 2023-03-20 00:57:26,296 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/ebe80fa3aad044918d8b4c178d16e95d 2023-03-20 00:57:26,296 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/b7114eba9aa447179319b4bb94d5ab1f to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/b7114eba9aa447179319b4bb94d5ab1f 2023-03-20 00:57:26,298 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/b3b3403c3f284f4f84180bfc48f5194f to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/b3b3403c3f284f4f84180bfc48f5194f 2023-03-20 00:57:26,300 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/a2daf73da87a4fd19d1d2912db845805 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/a2daf73da87a4fd19d1d2912db845805 2023-03-20 00:57:26,300 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/f4ef8b79b1084bd292623ed21d268edc as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/f4ef8b79b1084bd292623ed21d268edc 2023-03-20 00:57:26,301 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/2508c9659d304d3287e838f7975d4e3e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/2508c9659d304d3287e838f7975d4e3e 2023-03-20 00:57:26,304 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/8ffea4882e8242bbb1e414ea261b5707 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/8ffea4882e8242bbb1e414ea261b5707 2023-03-20 00:57:26,304 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/f4ef8b79b1084bd292623ed21d268edc, entries=150, sequenceid=101, filesize=11.6 K 2023-03-20 00:57:26,305 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/cfa96489f21f4296801226298e8d1b9c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/a5d74c88ab6b4969a382bfb0bb0ea15b, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/132da6828d6f48518df807f41d05b1df, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/b64dc294974c46e6aeaed195c989b402, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/06b8272673494d96a6b9f6ce13dc79d9, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/3be9559f58c24116b979e74cd14f5474] to archive 2023-03-20 00:57:26,306 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 00:57:26,306 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/7f9d98ce5d2943c4a80820d80c66fbe9 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/7f9d98ce5d2943c4a80820d80c66fbe9 2023-03-20 00:57:26,312 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/cfa96489f21f4296801226298e8d1b9c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/cfa96489f21f4296801226298e8d1b9c 2023-03-20 00:57:26,314 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/a5d74c88ab6b4969a382bfb0bb0ea15b to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/a5d74c88ab6b4969a382bfb0bb0ea15b 2023-03-20 00:57:26,314 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/7f9d98ce5d2943c4a80820d80c66fbe9, entries=150, sequenceid=101, filesize=11.6 K 2023-03-20 00:57:26,314 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/ebe80fa3aad044918d8b4c178d16e95d as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/ebe80fa3aad044918d8b4c178d16e95d 2023-03-20 00:57:26,315 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/132da6828d6f48518df807f41d05b1df to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/132da6828d6f48518df807f41d05b1df 2023-03-20 00:57:26,316 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/b64dc294974c46e6aeaed195c989b402 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/b64dc294974c46e6aeaed195c989b402 2023-03-20 00:57:26,317 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/06b8272673494d96a6b9f6ce13dc79d9 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/06b8272673494d96a6b9f6ce13dc79d9 2023-03-20 00:57:26,318 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/ebe80fa3aad044918d8b4c178d16e95d, entries=150, sequenceid=101, filesize=11.6 K 2023-03-20 00:57:26,318 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 718cd5d8d8fa4a8a2d92da64e6fb813f in 853ms, sequenceid=101, compaction requested=false 2023-03-20 00:57:26,318 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:26,318 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:26,319 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:26,319 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:26,319 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,319 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:26,319 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:26,319 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/3be9559f58c24116b979e74cd14f5474 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/3be9559f58c24116b979e74cd14f5474 2023-03-20 00:57:26,320 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:26,320 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:26,320 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:26,320 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:26,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:26,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59d75c26[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:26,321 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:26,321 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:26,321 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:26,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:26,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:26,321 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:26,321 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:26,321 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:26,322 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,322 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:26,322 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:26,322 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:26,322 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:26,322 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:26,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:26,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:26,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:26,324 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:26,324 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:26,324 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:26,324 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:26,324 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:26,325 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:26,325 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:26,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:26,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:26,325 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:26,326 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,326 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:26,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:26,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:26,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:26,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:26,327 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,327 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:26,327 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:26,327 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,327 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:26,327 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:26,327 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:26,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:26,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:26,327 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:26,327 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:26,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:26,327 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:26,328 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:26,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:26,328 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:26,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:26,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:26,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:26,428 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:26,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:26,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:26,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:26,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:26,432 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:26,432 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:26,432 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:26,432 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:26,432 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:26,432 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:26,433 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:26,433 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:26,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:26,433 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,434 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:26,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:26,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:26,434 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:26,434 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:26,434 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:26,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:26,435 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:26,435 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:26,438 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:26,438 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:26,438 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:26,438 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:26,438 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 718cd5d8d8fa4a8a2d92da64e6fb813f 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-20 00:57:26,438 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=A 2023-03-20 00:57:26,438 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:26,438 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=B 2023-03-20 00:57:26,439 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:26,439 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=C 2023-03-20 00:57:26,439 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:26,455 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/e310e63ce8a2403580b0463c8c3652ed 2023-03-20 00:57:26,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:26,692 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:26,708 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:26,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273906703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:26,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273906704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,710 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:26,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273906705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:26,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273906707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:26,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273906708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:26,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273906809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:26,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273906811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:26,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273906811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:26,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273906816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,819 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:26,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273906816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,864 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/f9ff631ba25d43e29c208a9768745611 2023-03-20 00:57:26,877 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/3fa5ca8b189c4482a8131e881c5110fb 2023-03-20 00:57:26,890 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/e310e63ce8a2403580b0463c8c3652ed as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/e310e63ce8a2403580b0463c8c3652ed 2023-03-20 00:57:26,894 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/e310e63ce8a2403580b0463c8c3652ed, entries=150, sequenceid=120, filesize=11.6 K 2023-03-20 00:57:26,895 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/f9ff631ba25d43e29c208a9768745611 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/f9ff631ba25d43e29c208a9768745611 2023-03-20 00:57:26,899 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/f9ff631ba25d43e29c208a9768745611, entries=150, sequenceid=120, filesize=11.6 K 2023-03-20 00:57:26,900 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/3fa5ca8b189c4482a8131e881c5110fb as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/3fa5ca8b189c4482a8131e881c5110fb 2023-03-20 00:57:26,905 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/3fa5ca8b189c4482a8131e881c5110fb, entries=150, sequenceid=120, filesize=11.6 K 2023-03-20 00:57:26,906 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 718cd5d8d8fa4a8a2d92da64e6fb813f in 468ms, sequenceid=120, compaction requested=true 2023-03-20 00:57:26,906 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:26,906 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:26,906 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:26,906 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:26,906 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,906 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:26,906 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:26,907 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,907 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:26,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:26,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:26,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:26,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:26,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:26,909 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:26,909 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:26,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:26,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:26,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3411f012[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:26,910 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:26,910 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:26,910 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:26,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:26,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:26,911 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:26,911 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:26,911 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:26,912 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,912 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:26,912 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:26,912 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:26,912 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:26,912 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:26,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:26,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:26,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:26,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:26,915 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:26,915 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:26,915 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:26,915 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:26,915 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:26,916 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:26,916 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:26,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:26,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:26,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:26,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:26,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:26,917 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:26,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:26,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:26,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:26,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:26,918 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:26,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:26,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:26,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,920 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:26,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:26,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,925 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:26,925 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:26,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:26,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:26,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:26,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:26,926 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,926 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:26,926 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:26,926 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:26,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:26,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:26,926 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:26,926 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:26,926 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:26,926 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:26,926 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:26,926 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:27,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:27,020 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 718cd5d8d8fa4a8a2d92da64e6fb813f 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-20 00:57:27,020 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=A 2023-03-20 00:57:27,020 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:27,020 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=B 2023-03-20 00:57:27,020 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:27,020 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=C 2023-03-20 00:57:27,020 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:27,026 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:27,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:27,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:27,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:27,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:27,036 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:27,036 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:27,036 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:27,036 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:27,037 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,037 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:27,037 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:27,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:27,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:27,037 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:27,038 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,038 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:27,038 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/a3f403b0218145ddbcc6a1b50c80d8ba 2023-03-20 00:57:27,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:27,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:27,040 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:27,040 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:27,040 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:27,040 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:27,040 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:27,040 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:27,040 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:27,040 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:27,040 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:27,041 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:27,041 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:27,041 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:27,041 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:27,041 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:27,041 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,041 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:27,041 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:27,042 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:27,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:27,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:27,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:27,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:27,046 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,046 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:27,046 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:27,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:27,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25de4893[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:27,046 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:27,047 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,047 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,047 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:27,047 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:27,047 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:27,048 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:27,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273907037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,048 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,048 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:27,050 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:27,050 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:27,050 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,050 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:27,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:27,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:27,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:27,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:27,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:27,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:27,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:27,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:27,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:27,053 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:27,053 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:27,053 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:27,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:27,053 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:27,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273907040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,053 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:27,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:27,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273907040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:27,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273907047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:27,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273907048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,054 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/823830463d4b4386bae7ff42c6be87ba 2023-03-20 00:57:27,054 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,054 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,054 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:27,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:27,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:27,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,055 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:27,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:27,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:27,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:27,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:27,056 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,058 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:27,058 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:27,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:27,058 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,059 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:27,059 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:27,059 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,059 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,059 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,059 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:27,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:27,059 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:27,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:27,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:27,059 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:27,059 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:27,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:27,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:27,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273907149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,159 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:27,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:27,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273907154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:27,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:27,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273907154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:27,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273907154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:27,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273907154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:27,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:27,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:27,162 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:27,162 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:27,162 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:27,162 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:27,162 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,162 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:27,163 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,163 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:27,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:27,163 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,163 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:27,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:27,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:27,164 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:27,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:27,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:27,164 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:27,164 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:27,164 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:27,164 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:27,164 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:27,164 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:27,164 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:27,164 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:27,164 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:27,164 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:27,164 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:27,164 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,164 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:27,164 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:27,165 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,165 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:27,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:27,165 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,166 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:27,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:27,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:27,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:27,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:27,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:27,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@212c13ed[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:27,167 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:27,167 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,167 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,167 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:27,167 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:27,167 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,167 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:27,167 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:27,168 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:27,168 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:27,168 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,168 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:27,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:27,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:27,170 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:27,170 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:27,170 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:27,170 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:27,170 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:27,172 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,172 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,172 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:27,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:27,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:27,173 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:27,174 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,175 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,175 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:27,176 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:27,175 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:27,176 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:27,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:27,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:27,176 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:27,176 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:27,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:27,176 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:27,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:27,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:27,176 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,176 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,176 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:27,176 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:27,276 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:27,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:27,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:27,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:27,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:27,280 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:27,280 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:27,280 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:27,280 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:27,281 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,281 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:27,281 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,282 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:27,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:27,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:27,282 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,282 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:27,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:27,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:27,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:27,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:27,283 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:27,283 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:27,283 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:27,283 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:27,283 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:27,288 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:27,288 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:27,288 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:27,288 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:27,288 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:27,288 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:27,288 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:27,288 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:27,288 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,288 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:27,288 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:27,298 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,298 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:27,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:27,298 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,298 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:27,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:27,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:27,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:27,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:27,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:27,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42794d82[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:27,300 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:27,300 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,300 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,300 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:27,301 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:27,301 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:27,301 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,301 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:27,301 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,301 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:27,301 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:27,301 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:27,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:27,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:27,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:27,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:27,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:27,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:27,304 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:27,304 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:27,304 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:27,304 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:27,304 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:27,307 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,307 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:27,307 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:27,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:27,307 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:27,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:27,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:27,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:27,308 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:27,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:27,310 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:27,310 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:27,310 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:27,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:27,310 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:27,310 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:27,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:27,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:27,310 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:27,310 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:27,310 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:27,311 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:27,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:27,311 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,311 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,311 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:27,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273907358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,364 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:27,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273907360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:27,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273907361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,365 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:27,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273907361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:27,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273907362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,411 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:27,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:27,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:27,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:27,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:27,414 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:27,414 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:27,414 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:27,414 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:27,414 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,414 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:27,415 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:27,415 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:27,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:27,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:27,416 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,416 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:27,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:27,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:27,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:27,417 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:27,417 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:27,417 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:27,417 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:27,417 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:27,418 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:27,418 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:27,418 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:27,418 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:27,418 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:27,418 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:27,418 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:27,418 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:27,418 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,418 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:27,418 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:27,419 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,419 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:27,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:27,419 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,419 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:27,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:27,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:27,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:27,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:27,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:27,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79d3b4bf[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:27,421 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:27,422 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,423 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,423 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:27,423 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,423 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:27,423 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:27,423 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:27,423 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,423 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:27,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:27,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:27,424 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:27,424 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:27,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:27,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:27,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:27,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:27,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:27,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:27,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:27,426 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:27,426 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:27,426 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:27,426 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:27,426 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:27,427 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,427 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:27,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:27,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,427 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:27,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:27,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:27,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:27,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:27,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:27,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:27,429 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,430 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,432 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:27,432 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:27,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:27,432 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:27,432 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:27,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:27,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:27,432 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:27,432 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:27,432 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,432 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,432 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:27,432 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,432 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:27,432 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:27,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:27,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:27,490 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/7879aeee113e4986b94baf09d505d3c6 2023-03-20 00:57:27,493 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/a3f403b0218145ddbcc6a1b50c80d8ba as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/a3f403b0218145ddbcc6a1b50c80d8ba 2023-03-20 00:57:27,496 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/a3f403b0218145ddbcc6a1b50c80d8ba, entries=200, sequenceid=136, filesize=14.0 K 2023-03-20 00:57:27,497 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/823830463d4b4386bae7ff42c6be87ba as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/823830463d4b4386bae7ff42c6be87ba 2023-03-20 00:57:27,500 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/823830463d4b4386bae7ff42c6be87ba, entries=150, sequenceid=136, filesize=11.7 K 2023-03-20 00:57:27,501 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/7879aeee113e4986b94baf09d505d3c6 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/7879aeee113e4986b94baf09d505d3c6 2023-03-20 00:57:27,504 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/7879aeee113e4986b94baf09d505d3c6, entries=150, sequenceid=136, filesize=11.7 K 2023-03-20 00:57:27,504 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 718cd5d8d8fa4a8a2d92da64e6fb813f in 484ms, sequenceid=136, compaction requested=true 2023-03-20 00:57:27,504 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:27,504 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:27,504 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 00:57:27,505 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:27,505 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 00:57:27,505 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:27,506 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47850 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 00:57:27,506 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50240 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 00:57:27,506 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 718cd5d8d8fa4a8a2d92da64e6fb813f/B is initiating minor compaction (all files) 2023-03-20 00:57:27,506 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 718cd5d8d8fa4a8a2d92da64e6fb813f/A is initiating minor compaction (all files) 2023-03-20 00:57:27,506 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 718cd5d8d8fa4a8a2d92da64e6fb813f/B in TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:27,506 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 718cd5d8d8fa4a8a2d92da64e6fb813f/A in TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:27,507 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/b9cdbc543018454d9a6aa4e70d5b224e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/7f9d98ce5d2943c4a80820d80c66fbe9, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/f9ff631ba25d43e29c208a9768745611, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/823830463d4b4386bae7ff42c6be87ba] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp, totalSize=46.7 K 2023-03-20 00:57:27,507 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/69129982a8c34a629d9eba3c2f114346, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/f4ef8b79b1084bd292623ed21d268edc, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/e310e63ce8a2403580b0463c8c3652ed, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/a3f403b0218145ddbcc6a1b50c80d8ba] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp, totalSize=49.1 K 2023-03-20 00:57:27,507 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting b9cdbc543018454d9a6aa4e70d5b224e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1679273842649 2023-03-20 00:57:27,507 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 69129982a8c34a629d9eba3c2f114346, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1679273842649 2023-03-20 00:57:27,507 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 7f9d98ce5d2943c4a80820d80c66fbe9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1679273844928 2023-03-20 00:57:27,507 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting f4ef8b79b1084bd292623ed21d268edc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1679273844928 2023-03-20 00:57:27,507 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting f9ff631ba25d43e29c208a9768745611, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1679273845562 2023-03-20 00:57:27,507 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting e310e63ce8a2403580b0463c8c3652ed, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1679273845562 2023-03-20 00:57:27,507 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 823830463d4b4386bae7ff42c6be87ba, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1679273846696 2023-03-20 00:57:27,508 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting a3f403b0218145ddbcc6a1b50c80d8ba, keycount=200, bloomtype=ROW, size=14.0 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1679273846696 2023-03-20 00:57:27,532 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:27,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:27,533 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 718cd5d8d8fa4a8a2d92da64e6fb813f#A#compaction#363 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:27,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:27,536 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 718cd5d8d8fa4a8a2d92da64e6fb813f#B#compaction#364 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:27,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:27,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:27,541 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:27,541 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:27,541 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:27,541 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:27,542 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,542 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:27,542 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:27,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:27,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:27,543 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:27,543 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,543 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:27,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:27,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:27,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:27,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:27,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:27,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:27,554 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:27,554 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:27,554 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:27,558 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:27,558 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:27,558 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:27,558 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:27,558 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 718cd5d8d8fa4a8a2d92da64e6fb813f 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-20 00:57:27,558 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=A 2023-03-20 00:57:27,558 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:27,558 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=B 2023-03-20 00:57:27,559 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:27,559 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=C 2023-03-20 00:57:27,559 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:27,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:27,670 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:27,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:27,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273907682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:27,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273907686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:27,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273907690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,704 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:27,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273907694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,704 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:27,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273907694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:27,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273907795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:27,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273907798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:27,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273907799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:27,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273907805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:27,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273907806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:27,952 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/156c5ade8e0d404eb28d5dba0469d774 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/156c5ade8e0d404eb28d5dba0469d774 2023-03-20 00:57:27,959 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 718cd5d8d8fa4a8a2d92da64e6fb813f/B of 718cd5d8d8fa4a8a2d92da64e6fb813f into 156c5ade8e0d404eb28d5dba0469d774(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:27,959 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:27,959 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f., storeName=718cd5d8d8fa4a8a2d92da64e6fb813f/B, priority=12, startTime=1679273847505; duration=0sec 2023-03-20 00:57:27,959 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:27,959 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 00:57:27,961 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47850 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 00:57:27,961 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 718cd5d8d8fa4a8a2d92da64e6fb813f/C is initiating minor compaction (all files) 2023-03-20 00:57:27,961 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 718cd5d8d8fa4a8a2d92da64e6fb813f/C in TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:27,961 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/132109c216d34c63ba19e48566e93b65, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/ebe80fa3aad044918d8b4c178d16e95d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/3fa5ca8b189c4482a8131e881c5110fb, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/7879aeee113e4986b94baf09d505d3c6] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp, totalSize=46.7 K 2023-03-20 00:57:27,964 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 132109c216d34c63ba19e48566e93b65, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1679273842649 2023-03-20 00:57:27,964 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting ebe80fa3aad044918d8b4c178d16e95d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1679273844928 2023-03-20 00:57:27,965 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 3fa5ca8b189c4482a8131e881c5110fb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1679273845562 2023-03-20 00:57:27,965 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 7879aeee113e4986b94baf09d505d3c6, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1679273846696 2023-03-20 00:57:27,965 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/ef707cfc0d5c42f5840f6663fd4ef206 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/ef707cfc0d5c42f5840f6663fd4ef206 2023-03-20 00:57:27,969 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/fa3a3c378c6040918c085aada9947f96 2023-03-20 00:57:27,973 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 718cd5d8d8fa4a8a2d92da64e6fb813f/A of 718cd5d8d8fa4a8a2d92da64e6fb813f into ef707cfc0d5c42f5840f6663fd4ef206(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:27,973 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:27,974 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f., storeName=718cd5d8d8fa4a8a2d92da64e6fb813f/A, priority=12, startTime=1679273847504; duration=0sec 2023-03-20 00:57:27,974 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:28,011 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:28,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:28,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273908002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:28,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273908002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,013 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 718cd5d8d8fa4a8a2d92da64e6fb813f#C#compaction#367 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:28,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:28,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273908012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273908000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:28,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273908017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,037 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/45c9b2fef5e74443951a7e342b8777d3 2023-03-20 00:57:28,064 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/fd895a9e8fca4764bc618c67a1042ef2 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/fd895a9e8fca4764bc618c67a1042ef2 2023-03-20 00:57:28,072 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 718cd5d8d8fa4a8a2d92da64e6fb813f/C of 718cd5d8d8fa4a8a2d92da64e6fb813f into fd895a9e8fca4764bc618c67a1042ef2(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:28,073 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:28,073 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f., storeName=718cd5d8d8fa4a8a2d92da64e6fb813f/C, priority=12, startTime=1679273847505; duration=0sec 2023-03-20 00:57:28,073 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:28,080 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/9bdcddf4aa6b4dabbf23e0811d9c54ff 2023-03-20 00:57:28,083 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/fa3a3c378c6040918c085aada9947f96 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/fa3a3c378c6040918c085aada9947f96 2023-03-20 00:57:28,087 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/fa3a3c378c6040918c085aada9947f96, entries=150, sequenceid=152, filesize=11.8 K 2023-03-20 00:57:28,087 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/45c9b2fef5e74443951a7e342b8777d3 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/45c9b2fef5e74443951a7e342b8777d3 2023-03-20 00:57:28,093 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/45c9b2fef5e74443951a7e342b8777d3, entries=150, sequenceid=152, filesize=11.8 K 2023-03-20 00:57:28,094 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/9bdcddf4aa6b4dabbf23e0811d9c54ff as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/9bdcddf4aa6b4dabbf23e0811d9c54ff 2023-03-20 00:57:28,099 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/9bdcddf4aa6b4dabbf23e0811d9c54ff, entries=150, sequenceid=152, filesize=11.8 K 2023-03-20 00:57:28,100 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 718cd5d8d8fa4a8a2d92da64e6fb813f in 541ms, sequenceid=152, compaction requested=false 2023-03-20 00:57:28,100 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:28,100 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:28,100 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:28,100 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:28,100 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,100 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:28,100 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:28,101 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,101 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:28,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:28,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:28,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:28,102 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:28,102 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:28,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:28,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:28,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:28,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:28,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e1ef1c6[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:28,105 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:28,105 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:28,106 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:28,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:28,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:28,106 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:28,106 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:28,106 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:28,106 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,106 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:28,107 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:28,107 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:28,107 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:28,107 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:28,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:28,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:28,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:28,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:28,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:28,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:28,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:28,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:28,110 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:28,110 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:28,110 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:28,110 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:28,110 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:28,118 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:28,118 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:28,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:28,118 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:28,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:28,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:28,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:28,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:28,119 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:28,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:28,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:28,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:28,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:28,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:28,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:28,121 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:28,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:28,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,125 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:28,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:28,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:28,125 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,125 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:28,125 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:28,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:28,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:28,125 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:28,125 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:28,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:28,125 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,125 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:28,125 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:28,125 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:28,126 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:28,126 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:28,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:28,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:28,226 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:28,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:28,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:28,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:28,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:28,230 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:28,230 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:28,230 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:28,230 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:28,230 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:28,230 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:28,231 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,231 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:28,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:28,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:28,231 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,231 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:28,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:28,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:28,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:28,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:28,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:28,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:28,237 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:28,237 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:28,237 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:28,237 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:28,237 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:28,237 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:28,237 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:28,237 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:28,237 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 718cd5d8d8fa4a8a2d92da64e6fb813f 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-20 00:57:28,237 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=A 2023-03-20 00:57:28,237 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:28,238 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=B 2023-03-20 00:57:28,238 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:28,238 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=C 2023-03-20 00:57:28,238 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:28,251 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/4be38a2b23c74bd2a4c9dc97518e0cd3 2023-03-20 00:57:28,286 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/9053f60d3ffc425ca9a91a1443efecbd 2023-03-20 00:57:28,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:28,324 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:28,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:28,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273908333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:28,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273908336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:28,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273908337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:28,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273908338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:28,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273908338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:28,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273908440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,444 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:28,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273908440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,447 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:28,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273908445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:28,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273908445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:28,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273908445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:28,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273908645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:28,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273908645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:28,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273908648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:28,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273908649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:28,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273908651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,698 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/c1065023417c45a7810279a25a1e566b 2023-03-20 00:57:28,702 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/4be38a2b23c74bd2a4c9dc97518e0cd3 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/4be38a2b23c74bd2a4c9dc97518e0cd3 2023-03-20 00:57:28,705 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/4be38a2b23c74bd2a4c9dc97518e0cd3, entries=150, sequenceid=171, filesize=11.8 K 2023-03-20 00:57:28,706 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/9053f60d3ffc425ca9a91a1443efecbd as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/9053f60d3ffc425ca9a91a1443efecbd 2023-03-20 00:57:28,709 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/9053f60d3ffc425ca9a91a1443efecbd, entries=150, sequenceid=171, filesize=11.8 K 2023-03-20 00:57:28,710 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/c1065023417c45a7810279a25a1e566b as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/c1065023417c45a7810279a25a1e566b 2023-03-20 00:57:28,715 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/c1065023417c45a7810279a25a1e566b, entries=150, sequenceid=171, filesize=11.8 K 2023-03-20 00:57:28,716 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 718cd5d8d8fa4a8a2d92da64e6fb813f in 479ms, sequenceid=171, compaction requested=true 2023-03-20 00:57:28,716 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:28,716 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:28,716 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:28,716 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:28,716 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,718 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:28,718 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:28,719 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,720 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:28,720 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:28,720 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:28,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:28,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:28,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:28,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fec3c93[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:28,729 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:28,730 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:28,741 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:28,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:28,742 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:28,742 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:28,742 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:28,743 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:28,743 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:28,743 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:28,760 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,760 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:28,760 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:28,760 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:28,760 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:28,760 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:28,760 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:28,760 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:28,761 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:28,761 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:28,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:28,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:28,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:28,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:28,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:28,761 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:28,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:28,762 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:28,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:28,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:28,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:28,763 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:28,764 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,764 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:28,764 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:28,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:28,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:28,765 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:28,765 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:28,765 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:28,765 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:28,765 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:28,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:28,765 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,765 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:28,765 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:28,765 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:28,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:28,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:28,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:28,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:28,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:28,865 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:28,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:28,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:28,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:28,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:28,869 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:28,869 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:28,869 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:28,869 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:28,870 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:28,870 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:28,870 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:28,870 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:28,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:28,871 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,871 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:28,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:28,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:28,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:28,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:28,871 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:28,872 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:28,872 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:28,872 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:28,872 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:28,872 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:28,872 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:28,872 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:28,872 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:28,872 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:28,872 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 718cd5d8d8fa4a8a2d92da64e6fb813f 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-20 00:57:28,872 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=A 2023-03-20 00:57:28,872 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:28,873 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=B 2023-03-20 00:57:28,873 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:28,873 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=C 2023-03-20 00:57:28,873 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:28,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:28,956 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:28,976 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:28,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273908968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:28,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273908973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:28,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273908974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:28,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273908975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:28,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:28,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273908976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:29,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:29,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273909077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273909077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:29,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273909081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:29,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273909081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:29,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273909081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:29,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273909284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:29,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273909284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:29,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273909284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:29,290 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:29,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273909285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273909285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,291 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/f6b265cbd23b452fbe148fe2fcc5137c 2023-03-20 00:57:29,308 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/f178f899704446e883a8b1df59589743 2023-03-20 00:57:29,326 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/4b5d4373b9c0478f9c90f117a164bf57 2023-03-20 00:57:29,329 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/f6b265cbd23b452fbe148fe2fcc5137c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/f6b265cbd23b452fbe148fe2fcc5137c 2023-03-20 00:57:29,334 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/f6b265cbd23b452fbe148fe2fcc5137c, entries=150, sequenceid=187, filesize=11.8 K 2023-03-20 00:57:29,335 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/f178f899704446e883a8b1df59589743 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/f178f899704446e883a8b1df59589743 2023-03-20 00:57:29,346 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/f178f899704446e883a8b1df59589743, entries=150, sequenceid=187, filesize=11.8 K 2023-03-20 00:57:29,347 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/4b5d4373b9c0478f9c90f117a164bf57 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/4b5d4373b9c0478f9c90f117a164bf57 2023-03-20 00:57:29,351 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/4b5d4373b9c0478f9c90f117a164bf57, entries=150, sequenceid=187, filesize=11.8 K 2023-03-20 00:57:29,352 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 718cd5d8d8fa4a8a2d92da64e6fb813f in 480ms, sequenceid=187, compaction requested=true 2023-03-20 00:57:29,352 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:29,352 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:29,352 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:29,353 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:29,356 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,356 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:29,356 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:29,357 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:29,358 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:29,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:29,358 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:29,358 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:29,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:29,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:29,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:29,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:29,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:29,359 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:29,359 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:29,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@677179b1[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:29,360 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:29,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:29,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:29,360 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:29,360 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:29,360 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:29,360 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,360 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:29,361 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:29,361 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:29,361 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:29,361 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:29,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:29,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:29,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:29,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:29,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:29,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:29,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:29,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:29,364 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:29,364 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:29,364 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:29,364 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:29,364 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:29,365 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:29,365 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:29,365 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:29,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:29,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:29,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:29,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:29,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:29,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:29,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:29,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:29,366 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:29,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:29,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:29,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:29,369 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,383 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,383 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:29,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:29,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:29,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:29,383 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:29,383 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:29,384 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:29,384 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:29,383 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:29,384 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:29,384 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,384 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:29,384 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:29,384 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:29,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:29,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:29,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:29,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:29,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:29,484 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:29,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:29,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:29,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:29,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:29,490 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:29,490 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:29,490 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:29,490 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:29,491 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:29,491 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:29,492 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,492 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,492 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:29,492 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:29,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:29,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:29,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:29,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:29,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:29,493 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:29,499 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:29,499 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:29,499 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:29,499 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:29,499 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:29,502 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:29,502 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:29,502 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:29,502 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:29,502 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 718cd5d8d8fa4a8a2d92da64e6fb813f 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-20 00:57:29,503 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=A 2023-03-20 00:57:29,503 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:29,503 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=B 2023-03-20 00:57:29,503 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:29,503 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=C 2023-03-20 00:57:29,503 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:29,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:29,591 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:29,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:29,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273909609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:29,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273909610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,622 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:29,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273909615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,625 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:29,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273909615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,626 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:29,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273909615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:29,726 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:29,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273909718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273909719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,731 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:29,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273909723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,731 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:29,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273909727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:29,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273909727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,937 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:29,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273909928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,937 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:29,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273909928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,939 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:29,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273909932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,939 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:29,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273909932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,939 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:29,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273909933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:29,976 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/ef96b31236bd427cbaa1b61babe61531 2023-03-20 00:57:30,246 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:30,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273910240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:30,246 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:30,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273910241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:30,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:30,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273910242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:30,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:30,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273910242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:30,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:30,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273910242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:30,432 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/1948a5cb8f8e4a51ace11109d49cff54 2023-03-20 00:57:30,453 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/ea750c050a694f9985950023a0675b86 2023-03-20 00:57:30,461 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/ef96b31236bd427cbaa1b61babe61531 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/ef96b31236bd427cbaa1b61babe61531 2023-03-20 00:57:30,465 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/ef96b31236bd427cbaa1b61babe61531, entries=150, sequenceid=203, filesize=11.8 K 2023-03-20 00:57:30,466 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/1948a5cb8f8e4a51ace11109d49cff54 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/1948a5cb8f8e4a51ace11109d49cff54 2023-03-20 00:57:30,474 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/1948a5cb8f8e4a51ace11109d49cff54, entries=150, sequenceid=203, filesize=11.8 K 2023-03-20 00:57:30,474 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/ea750c050a694f9985950023a0675b86 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/ea750c050a694f9985950023a0675b86 2023-03-20 00:57:30,485 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/ea750c050a694f9985950023a0675b86, entries=150, sequenceid=203, filesize=11.8 K 2023-03-20 00:57:30,485 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 718cd5d8d8fa4a8a2d92da64e6fb813f in 983ms, sequenceid=203, compaction requested=true 2023-03-20 00:57:30,493 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:30,493 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:30,493 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:30,493 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:30,493 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:30,493 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:30,493 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:30,494 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:30,494 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:30,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:30,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:30,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:30,495 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:30,495 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:30,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:30,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:30,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:30,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:30,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:30,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:30,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e35fac[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:30,498 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:30,498 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:30,499 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:30,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:30,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:30,499 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:30,499 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:30,499 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:30,500 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:30,500 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:30,500 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:30,500 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:30,500 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:30,501 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:30,501 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:30,501 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:30,501 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:30,501 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:30,501 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:30,502 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:30,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:30,502 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:30,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:30,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:30,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:30,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:30,502 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:30,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:30,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:30,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:30,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:30,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:30,503 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:30,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:30,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:30,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:30,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:30,504 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:30,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:30,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:30,508 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:30,508 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:30,508 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:30,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:30,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:30,508 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:30,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:30,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:30,509 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:30,509 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:30,508 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:30,508 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:30,510 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:30,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:30,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:30,510 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:30,510 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:30,510 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:30,609 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:30,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:30,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:30,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:30,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:30,614 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:30,614 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:30,614 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:30,614 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:30,615 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:30,615 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:30,615 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:30,615 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:30,615 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:30,616 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:30,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:30,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:30,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:30,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:30,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:30,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:30,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:30,626 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:30,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:30,626 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:30,626 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:30,626 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:30,630 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:30,630 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:30,630 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:30,630 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:30,630 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 718cd5d8d8fa4a8a2d92da64e6fb813f 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-20 00:57:30,630 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=A 2023-03-20 00:57:30,631 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:30,631 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=B 2023-03-20 00:57:30,631 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:30,631 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=C 2023-03-20 00:57:30,631 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:30,645 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/1137b78030044abf9aa3b39d2a2fcef7 2023-03-20 00:57:30,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:30,757 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:30,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:30,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273910775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:30,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:30,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273910776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:30,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:30,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273910777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:30,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:30,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273910778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:30,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:30,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273910779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:30,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:30,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273910890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:30,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:30,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273910890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:30,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:30,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273910891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:30,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:30,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273910891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:30,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:30,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273910892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,067 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/6324a30282fe41eeaa7f18ed7410f10b 2023-03-20 00:57:31,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:31,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:31,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273911097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273911097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:31,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273911097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,104 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:31,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273911098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:31,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52350 deadline: 1679273911104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:31,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273911405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:31,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273911405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:31,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273911406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:31,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52300 deadline: 1679273911407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,418 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:31,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273911412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,477 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/a0a43deccb39400f864456740015f3d3 2023-03-20 00:57:31,484 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/1137b78030044abf9aa3b39d2a2fcef7 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/1137b78030044abf9aa3b39d2a2fcef7 2023-03-20 00:57:31,487 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/1137b78030044abf9aa3b39d2a2fcef7, entries=150, sequenceid=219, filesize=11.8 K 2023-03-20 00:57:31,487 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/6324a30282fe41eeaa7f18ed7410f10b as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/6324a30282fe41eeaa7f18ed7410f10b 2023-03-20 00:57:31,491 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/6324a30282fe41eeaa7f18ed7410f10b, entries=150, sequenceid=219, filesize=11.8 K 2023-03-20 00:57:31,491 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/a0a43deccb39400f864456740015f3d3 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/a0a43deccb39400f864456740015f3d3 2023-03-20 00:57:31,495 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/a0a43deccb39400f864456740015f3d3, entries=150, sequenceid=219, filesize=11.8 K 2023-03-20 00:57:31,496 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 718cd5d8d8fa4a8a2d92da64e6fb813f in 866ms, sequenceid=219, compaction requested=true 2023-03-20 00:57:31,496 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:31,496 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:31,496 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:31,496 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:31,496 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,496 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:31,496 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:31,497 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,497 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:31,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:31,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:31,497 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:31,497 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:31,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:31,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:31,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:31,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:31,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:31,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5aaa0ed7[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:31,505 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:31,505 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:31,516 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:31,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:31,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:31,516 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:31,516 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,516 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:31,516 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:31,517 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:31,517 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:31,517 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,517 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:31,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:31,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:31,517 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:31,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:31,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:31,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:31,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:31,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:31,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:31,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:31,519 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:31,519 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:31,519 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:31,519 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:31,519 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:31,520 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:31,520 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:31,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:31,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:31,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:31,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:31,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:31,520 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:31,520 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:31,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:31,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:31,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:31,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:31,521 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:31,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:31,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,523 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:31,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:31,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:31,523 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:31,523 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:31,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:31,523 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:31,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:31,523 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:31,523 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:31,523 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:31,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:31,523 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:31,523 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:31,523 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:31,523 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:31,523 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:31,623 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:31,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:31,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:31,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:31,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:31,626 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:31,626 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:31,626 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:31,626 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:31,627 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:31,627 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:31,627 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,627 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:31,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:31,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:31,628 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,628 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:31,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:31,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:31,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:31,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:31,628 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:31,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:31,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:31,628 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:31,629 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:31,629 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:31,629 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:31,629 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:31,629 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:31,629 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:31,629 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 718cd5d8d8fa4a8a2d92da64e6fb813f 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-20 00:57:31,629 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=A 2023-03-20 00:57:31,629 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:31,629 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=B 2023-03-20 00:57:31,629 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:31,629 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=C 2023-03-20 00:57:31,629 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:31,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:31,918 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:31,944 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:31,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273911930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,951 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:31,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52320 deadline: 1679273911943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,955 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:31,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273911944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,955 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:31,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273911944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:31,955 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:31,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273911944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,035 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/7cb25caf8ff64766a1ad244162843727 2023-03-20 00:57:32,046 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:32,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273912045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:32,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273912052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:32,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52302 deadline: 1679273912055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:32,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273912056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:32,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52358 deadline: 1679273912056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:32,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273912249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:32,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273912255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:32,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52302 deadline: 1679273912264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:32,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273912264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:32,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52358 deadline: 1679273912264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,444 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/c47d83db1cbc4daa924c33d92debce6d 2023-03-20 00:57:32,454 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/2cba857c4890448fa4ae96c18e42f5d9 2023-03-20 00:57:32,457 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/7cb25caf8ff64766a1ad244162843727 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/7cb25caf8ff64766a1ad244162843727 2023-03-20 00:57:32,460 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/7cb25caf8ff64766a1ad244162843727, entries=150, sequenceid=235, filesize=11.8 K 2023-03-20 00:57:32,460 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/c47d83db1cbc4daa924c33d92debce6d as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/c47d83db1cbc4daa924c33d92debce6d 2023-03-20 00:57:32,463 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/c47d83db1cbc4daa924c33d92debce6d, entries=150, sequenceid=235, filesize=11.8 K 2023-03-20 00:57:32,463 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/2cba857c4890448fa4ae96c18e42f5d9 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/2cba857c4890448fa4ae96c18e42f5d9 2023-03-20 00:57:32,466 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/2cba857c4890448fa4ae96c18e42f5d9, entries=150, sequenceid=235, filesize=11.8 K 2023-03-20 00:57:32,466 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 718cd5d8d8fa4a8a2d92da64e6fb813f in 837ms, sequenceid=235, compaction requested=true 2023-03-20 00:57:32,467 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:32,467 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:32,467 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:32,467 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:32,467 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,467 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:32,467 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:32,468 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,468 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:32,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:32,468 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,468 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:32,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:32,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:32,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:32,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:32,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:32,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54acb9b5[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:32,469 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:32,469 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,470 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,470 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:32,470 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:32,470 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,470 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:32,470 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:32,470 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:32,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:32,470 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:32,471 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:32,471 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:32,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:32,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:32,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:32,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:32,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:32,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:32,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:32,472 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:32,473 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:32,473 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:32,473 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:32,473 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:32,473 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,473 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,473 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:32,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:32,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:32,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,474 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:32,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:32,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:32,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:32,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:32,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:32,475 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:32,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:32,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,477 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:32,477 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,477 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:32,477 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:32,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:32,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:32,477 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:32,478 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:32,478 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,478 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,478 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:32,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:32,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:32,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:32,478 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:32,478 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,478 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:32,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:32,571 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 718cd5d8d8fa4a8a2d92da64e6fb813f 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-20 00:57:32,572 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=A 2023-03-20 00:57:32,572 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:32,572 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=B 2023-03-20 00:57:32,572 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:32,572 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=C 2023-03-20 00:57:32,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:32,572 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:32,579 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:32,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:32,580 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/2e545e9bd09741cca18d97fdf7d08a0b 2023-03-20 00:57:32,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:32,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:32,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:32,588 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:32,588 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:32,588 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:32,588 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:32,589 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,589 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:32,589 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,589 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:32,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:32,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:32,590 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,590 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:32,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:32,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:32,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:32,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:32,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:32,614 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:32,614 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:32,614 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:32,618 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:32,618 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:32,618 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:32,618 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:32,618 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:32,618 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:32,618 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:32,618 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:32,618 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,618 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:32,618 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:32,619 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,619 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:32,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:32,620 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,620 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:32,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:32,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:32,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:32,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:32,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:32,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@387cee60[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:32,621 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:32,621 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,622 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,622 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:32,622 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:32,622 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:32,622 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,622 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:32,622 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:32,623 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:32,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273912610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,623 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:32,623 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:32,623 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:32,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52302 deadline: 1679273912611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,624 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:32,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:32,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:32,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:32,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:32,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:32,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:32,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:32,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:32,626 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:32,626 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:32,626 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:32,626 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:32,626 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:32,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:32,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273912612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:32,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52358 deadline: 1679273912613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:32,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273912621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,633 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,633 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:32,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:32,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:32,634 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:32,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:32,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:32,634 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:32,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:32,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:32,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,635 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:32,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:32,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,639 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,639 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:32,639 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:32,639 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:32,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:32,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:32,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:32,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:32,639 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:32,640 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:32,640 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:32,640 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:32,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:32,640 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,640 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:32,640 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,640 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,730 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:32,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52302 deadline: 1679273912724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,730 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:32,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273912724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,731 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:32,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52358 deadline: 1679273912729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,731 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:32,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273912729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:32,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273912730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,740 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:32,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:32,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:32,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:32,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:32,743 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:32,743 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:32,743 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:32,743 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:32,743 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,743 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:32,744 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:32,744 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:32,744 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,744 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:32,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:32,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:32,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:32,745 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:32,745 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:32,745 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:32,745 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:32,745 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:32,745 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:32,745 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:32,745 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:32,745 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:32,745 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:32,745 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:32,745 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:32,745 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:32,745 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,746 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:32,746 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:32,746 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,747 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,747 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,747 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:32,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:32,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:32,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:32,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:32,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:32,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:32,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:32,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@217bd8fc[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:32,751 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:32,751 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,751 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,751 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:32,752 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:32,752 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:32,752 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,752 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:32,752 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:32,752 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:32,752 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,752 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:32,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:32,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:32,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:32,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:32,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:32,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:32,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:32,755 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:32,755 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:32,755 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:32,755 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:32,755 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:32,756 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,756 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,756 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:32,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:32,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:32,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:32,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:32,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:32,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,756 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:32,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:32,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:32,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,758 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:32,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:32,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,760 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:32,760 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,760 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:32,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:32,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:32,760 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:32,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:32,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:32,760 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:32,760 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:32,760 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:32,760 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:32,760 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,760 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,760 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,760 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:32,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:32,860 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:32,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:32,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:32,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:32,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:32,865 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:32,865 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:32,865 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:32,865 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:32,866 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,866 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:32,866 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,866 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:32,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:32,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:32,866 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,866 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:32,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:32,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:32,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:32,867 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:32,867 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:32,867 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:32,867 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:32,867 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:32,867 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:32,868 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:32,868 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:32,868 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:32,868 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:32,868 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:32,868 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:32,868 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:32,868 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:32,868 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,868 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:32,868 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:32,869 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,869 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:32,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:32,869 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:32,869 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:32,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:32,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:32,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:32,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:32,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b2da6b2[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:32,871 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:32,871 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,871 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,871 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:32,872 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:32,872 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:32,872 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,872 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:32,872 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:32,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:32,872 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:32,872 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:32,872 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:32,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:32,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:32,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:32,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:32,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:32,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:32,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:32,875 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:32,876 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:32,876 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:32,876 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:32,876 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:32,876 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,876 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:32,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:32,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,877 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:32,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:32,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:32,877 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:32,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:32,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:32,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:32,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,878 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:32,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:32,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,879 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,880 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:32,880 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:32,880 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:32,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:32,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:32,880 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:32,880 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:32,880 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,880 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:32,880 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,880 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:32,880 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:32,880 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:32,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:32,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:32,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:32,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52302 deadline: 1679273912931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:32,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273912932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:32,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273912932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,942 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:32,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52358 deadline: 1679273912932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:32,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273912939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,980 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:32,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:32,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:32,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:32,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:32,983 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:32,983 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:32,984 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:32,984 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:32,984 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,984 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:32,984 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,985 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:32,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:32,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:32,985 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,985 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:32,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:32,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:32,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:32,985 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:32,986 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:32,986 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:32,986 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:32,986 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:32,986 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:32,986 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:32,986 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:32,986 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:32,986 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:32,986 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:32,986 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:32,986 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:32,986 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:32,986 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,986 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:32,986 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:32,988 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,988 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:32,988 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,988 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:32,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:32,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:32,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:32,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1afdd373[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:32,989 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:32,989 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,990 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,990 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:32,990 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,990 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:32,990 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:32,990 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:32,991 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:32,991 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:32,991 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,991 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:32,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:32,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:32,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:32,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:32,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:32,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:32,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:32,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:32,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:32,993 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:32,993 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:32,993 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:32,993 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:32,993 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:32,994 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,994 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:32,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:32,994 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:32,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:32,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:32,994 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:32,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:32,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:32,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:32,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:32,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,995 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:32,996 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/16c63abdcf0142f8ad4f91e6a1789f8e 2023-03-20 00:57:32,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:32,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,998 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,999 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:32,999 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:32,999 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:32,999 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,999 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:32,999 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:33,000 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:33,000 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:33,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:33,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:33,001 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:33,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:33,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:33,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:33,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:33,001 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:33,001 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:33,101 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:33,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:33,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:33,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:33,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:33,105 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:33,105 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:33,105 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:33,105 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:33,105 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:33,105 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:33,106 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,106 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:33,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:33,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:33,106 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,106 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:33,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:33,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:33,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:33,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:33,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:33,113 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:33,113 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:33,113 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:33,113 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:33,113 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:33,113 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:33,113 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:33,113 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:33,113 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:33,113 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:33,113 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:33,113 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:33,113 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:33,113 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,114 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:33,114 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:33,114 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:33,114 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:33,115 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:33,115 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:33,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:33,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45ad21a2[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:33,116 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:33,116 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:33,116 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:33,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:33,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:33,116 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,116 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:33,116 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:33,117 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:33,117 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:33,117 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,117 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:33,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:33,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:33,117 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:33,117 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:33,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:33,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:33,119 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:33,119 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:33,119 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:33,119 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:33,119 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:33,120 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:33,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:33,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:33,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:33,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:33,121 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:33,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:33,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:33,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,122 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:33,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:33,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,123 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:33,124 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:33,124 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:33,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:33,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:33,124 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:33,124 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,124 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:33,124 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:33,124 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,124 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:33,124 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:33,124 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:33,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:33,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:33,125 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:33,125 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:33,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:33,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:33,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:33,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:33,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:33,225 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:33,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:33,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:33,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:33,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:33,228 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:33,228 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:33,228 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:33,228 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:33,229 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:33,229 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:33,229 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:33,229 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:33,230 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,230 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:33,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:33,230 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:33,234 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:33,234 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:33,234 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:33,234 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:33,234 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:33,237 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:33,238 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:33,238 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:33,238 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:33,238 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:33,238 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:33,238 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:33,238 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:33,238 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,238 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:33,238 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:33,247 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,247 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:33,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:33,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:33,248 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:33,248 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:33,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:33,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:33,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:33,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:33,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:33,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a6ebdd3[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:33,249 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:33,249 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:33,250 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:33,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:33,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:33,250 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:33,250 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:33,250 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:33,250 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,250 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:33,250 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:33,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52302 deadline: 1679273913242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,250 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:33,250 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:33,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273913242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,250 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:33,250 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:33,251 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:33,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52358 deadline: 1679273913242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,251 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:33,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273913243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,251 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:33,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:33,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:33,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:33,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:33,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:33,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:33,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:33,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:33,254 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:33,254 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:33,254 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:33,254 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:33,254 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:33,262 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:33,262 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:33,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:33,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:33,262 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:33,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:33,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:33,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:33,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:33,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:33,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:33,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:33,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273913253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:33,265 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:33,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:33,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:33,266 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:33,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:33,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,269 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:33,269 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:33,269 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:33,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:33,270 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:33,270 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:33,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:33,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:33,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:33,270 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:33,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:33,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:33,270 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:33,270 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:33,270 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,270 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:33,270 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:33,270 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:33,370 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:33,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:33,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:33,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:33,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:33,373 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:33,373 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:33,373 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:33,373 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:33,374 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:33,374 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:33,374 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:33,374 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:33,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:33,374 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,374 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:33,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:33,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:33,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:33,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:33,375 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:33,375 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:33,375 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:33,375 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:33,375 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:33,375 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:33,376 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:33,376 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:33,376 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:33,376 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:33,376 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:33,376 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:33,376 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:33,376 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:33,376 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,376 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:33,376 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:33,377 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,377 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:33,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:33,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:33,377 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:33,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:33,377 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:33,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:33,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:33,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:33,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:33,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c30841a[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:33,380 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:33,380 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:33,381 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:33,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:33,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:33,381 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:33,381 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,381 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:33,381 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:33,381 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:33,382 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,382 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:33,382 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:33,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:33,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:33,382 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:33,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:33,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:33,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:33,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:33,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:33,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:33,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:33,383 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:33,384 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:33,384 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:33,384 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:33,384 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:33,384 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:33,384 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:33,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:33,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:33,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:33,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:33,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:33,384 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:33,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:33,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:33,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:33,385 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:33,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:33,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:33,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:33,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,386 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:33,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:33,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,387 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,387 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:33,387 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:33,387 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:33,387 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,387 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:33,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:33,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:33,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:33,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:33,387 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:33,388 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:33,388 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:33,387 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:33,388 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:33,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:33,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:33,388 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:33,416 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/d0235b61b1da46bc90529d6192184b66 2023-03-20 00:57:33,421 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/2e545e9bd09741cca18d97fdf7d08a0b as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/2e545e9bd09741cca18d97fdf7d08a0b 2023-03-20 00:57:33,424 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/2e545e9bd09741cca18d97fdf7d08a0b, entries=200, sequenceid=252, filesize=14.1 K 2023-03-20 00:57:33,425 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/16c63abdcf0142f8ad4f91e6a1789f8e as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/16c63abdcf0142f8ad4f91e6a1789f8e 2023-03-20 00:57:33,429 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/16c63abdcf0142f8ad4f91e6a1789f8e, entries=150, sequenceid=252, filesize=11.8 K 2023-03-20 00:57:33,430 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/d0235b61b1da46bc90529d6192184b66 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/d0235b61b1da46bc90529d6192184b66 2023-03-20 00:57:33,434 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/d0235b61b1da46bc90529d6192184b66, entries=150, sequenceid=252, filesize=11.8 K 2023-03-20 00:57:33,435 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 718cd5d8d8fa4a8a2d92da64e6fb813f in 864ms, sequenceid=252, compaction requested=true 2023-03-20 00:57:33,435 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:33,435 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:33,435 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-03-20 00:57:33,435 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-03-20 00:57:33,435 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:33,435 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:33,437 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 98766 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-03-20 00:57:33,437 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 718cd5d8d8fa4a8a2d92da64e6fb813f/A is initiating minor compaction (all files) 2023-03-20 00:57:33,437 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 96376 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-03-20 00:57:33,437 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 718cd5d8d8fa4a8a2d92da64e6fb813f/A in TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:33,437 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 718cd5d8d8fa4a8a2d92da64e6fb813f/B is initiating minor compaction (all files) 2023-03-20 00:57:33,437 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/ef707cfc0d5c42f5840f6663fd4ef206, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/fa3a3c378c6040918c085aada9947f96, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/4be38a2b23c74bd2a4c9dc97518e0cd3, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/f6b265cbd23b452fbe148fe2fcc5137c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/ef96b31236bd427cbaa1b61babe61531, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/1137b78030044abf9aa3b39d2a2fcef7, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/7cb25caf8ff64766a1ad244162843727, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/2e545e9bd09741cca18d97fdf7d08a0b] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp, totalSize=96.5 K 2023-03-20 00:57:33,437 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 718cd5d8d8fa4a8a2d92da64e6fb813f/B in TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:33,437 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/156c5ade8e0d404eb28d5dba0469d774, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/45c9b2fef5e74443951a7e342b8777d3, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/9053f60d3ffc425ca9a91a1443efecbd, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/f178f899704446e883a8b1df59589743, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/1948a5cb8f8e4a51ace11109d49cff54, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/6324a30282fe41eeaa7f18ed7410f10b, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/c47d83db1cbc4daa924c33d92debce6d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/16c63abdcf0142f8ad4f91e6a1789f8e] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp, totalSize=94.1 K 2023-03-20 00:57:33,437 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting ef707cfc0d5c42f5840f6663fd4ef206, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1679273846696 2023-03-20 00:57:33,437 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting fa3a3c378c6040918c085aada9947f96, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1679273847038 2023-03-20 00:57:33,438 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 156c5ade8e0d404eb28d5dba0469d774, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1679273846696 2023-03-20 00:57:33,438 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 4be38a2b23c74bd2a4c9dc97518e0cd3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1679273847687 2023-03-20 00:57:33,438 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 45c9b2fef5e74443951a7e342b8777d3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1679273847038 2023-03-20 00:57:33,438 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 9053f60d3ffc425ca9a91a1443efecbd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1679273847687 2023-03-20 00:57:33,438 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting f6b265cbd23b452fbe148fe2fcc5137c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1679273848332 2023-03-20 00:57:33,438 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting f178f899704446e883a8b1df59589743, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1679273848332 2023-03-20 00:57:33,438 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting ef96b31236bd427cbaa1b61babe61531, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1679273848973 2023-03-20 00:57:33,439 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 1948a5cb8f8e4a51ace11109d49cff54, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1679273848973 2023-03-20 00:57:33,439 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 1137b78030044abf9aa3b39d2a2fcef7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1679273849607 2023-03-20 00:57:33,439 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 6324a30282fe41eeaa7f18ed7410f10b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1679273849607 2023-03-20 00:57:33,439 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 7cb25caf8ff64766a1ad244162843727, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1679273850771 2023-03-20 00:57:33,439 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting c47d83db1cbc4daa924c33d92debce6d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1679273850771 2023-03-20 00:57:33,439 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 2e545e9bd09741cca18d97fdf7d08a0b, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1679273851929 2023-03-20 00:57:33,439 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 16c63abdcf0142f8ad4f91e6a1789f8e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1679273851929 2023-03-20 00:57:33,470 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 718cd5d8d8fa4a8a2d92da64e6fb813f#A#compaction#387 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:33,482 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 718cd5d8d8fa4a8a2d92da64e6fb813f#B#compaction#388 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:33,488 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:33,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:33,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:33,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:33,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:33,496 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:33,496 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:33,496 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:33,496 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:33,496 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:33,496 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:33,497 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,497 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:33,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:33,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:33,498 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,498 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:33,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:33,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:33,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:33,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:33,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:33,502 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:33,502 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:33,502 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:33,502 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:33,502 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:33,508 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:33,509 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:33,509 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:33,509 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:33,510 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 718cd5d8d8fa4a8a2d92da64e6fb813f 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-20 00:57:33,510 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=A 2023-03-20 00:57:33,510 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:33,510 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=B 2023-03-20 00:57:33,510 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:33,510 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=C 2023-03-20 00:57:33,510 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:33,517 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/7759bbe5d17647a5974f880c3465954e as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/7759bbe5d17647a5974f880c3465954e 2023-03-20 00:57:33,520 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/44418d2305964c32b1ec416c15ebaf18 2023-03-20 00:57:33,522 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 718cd5d8d8fa4a8a2d92da64e6fb813f/B of 718cd5d8d8fa4a8a2d92da64e6fb813f into 7759bbe5d17647a5974f880c3465954e(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:33,522 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:33,522 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f., storeName=718cd5d8d8fa4a8a2d92da64e6fb813f/B, priority=8, startTime=1679273853435; duration=0sec 2023-03-20 00:57:33,522 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:33,522 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-03-20 00:57:33,534 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 96376 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-03-20 00:57:33,535 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 718cd5d8d8fa4a8a2d92da64e6fb813f/C is initiating minor compaction (all files) 2023-03-20 00:57:33,535 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 718cd5d8d8fa4a8a2d92da64e6fb813f/C in TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:33,535 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/fd895a9e8fca4764bc618c67a1042ef2, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/9bdcddf4aa6b4dabbf23e0811d9c54ff, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/c1065023417c45a7810279a25a1e566b, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/4b5d4373b9c0478f9c90f117a164bf57, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/ea750c050a694f9985950023a0675b86, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/a0a43deccb39400f864456740015f3d3, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/2cba857c4890448fa4ae96c18e42f5d9, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/d0235b61b1da46bc90529d6192184b66] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp, totalSize=94.1 K 2023-03-20 00:57:33,540 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting fd895a9e8fca4764bc618c67a1042ef2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1679273846696 2023-03-20 00:57:33,540 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 9bdcddf4aa6b4dabbf23e0811d9c54ff, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1679273847038 2023-03-20 00:57:33,541 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting c1065023417c45a7810279a25a1e566b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1679273847687 2023-03-20 00:57:33,541 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 4b5d4373b9c0478f9c90f117a164bf57, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1679273848332 2023-03-20 00:57:33,541 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting ea750c050a694f9985950023a0675b86, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1679273848973 2023-03-20 00:57:33,541 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting a0a43deccb39400f864456740015f3d3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1679273849607 2023-03-20 00:57:33,542 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 2cba857c4890448fa4ae96c18e42f5d9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1679273850771 2023-03-20 00:57:33,542 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting d0235b61b1da46bc90529d6192184b66, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1679273851929 2023-03-20 00:57:33,559 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 718cd5d8d8fa4a8a2d92da64e6fb813f#C#compaction#391 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:33,585 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/1f9704b490b64e90b96655c22b7d5cd6 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/1f9704b490b64e90b96655c22b7d5cd6 2023-03-20 00:57:33,589 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 718cd5d8d8fa4a8a2d92da64e6fb813f/C of 718cd5d8d8fa4a8a2d92da64e6fb813f into 1f9704b490b64e90b96655c22b7d5cd6(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:33,589 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:33,589 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f., storeName=718cd5d8d8fa4a8a2d92da64e6fb813f/C, priority=8, startTime=1679273853435; duration=0sec 2023-03-20 00:57:33,589 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:33,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:33,758 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:33,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:33,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273913778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:33,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52302 deadline: 1679273913779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:33,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52358 deadline: 1679273913779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:33,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273913786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:33,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273913787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,892 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/a4d62aa425a34b8d8498a4491da9bf6a as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/a4d62aa425a34b8d8498a4491da9bf6a 2023-03-20 00:57:33,894 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:33,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273913887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:33,895 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:33,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52302 deadline: 1679273913887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52358 deadline: 1679273913887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,895 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:33,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273913889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,896 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 718cd5d8d8fa4a8a2d92da64e6fb813f/A of 718cd5d8d8fa4a8a2d92da64e6fb813f into a4d62aa425a34b8d8498a4491da9bf6a(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:33,896 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:33,896 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f., storeName=718cd5d8d8fa4a8a2d92da64e6fb813f/A, priority=8, startTime=1679273853435; duration=0sec 2023-03-20 00:57:33,896 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:33,906 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:33,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273913898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,968 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/f3cd99ee2f344844be447bebc0574c04 2023-03-20 00:57:33,980 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/6bd281c0f57045a6ba17985bba36178d 2023-03-20 00:57:33,984 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/44418d2305964c32b1ec416c15ebaf18 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/44418d2305964c32b1ec416c15ebaf18 2023-03-20 00:57:33,987 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/44418d2305964c32b1ec416c15ebaf18, entries=150, sequenceid=267, filesize=11.9 K 2023-03-20 00:57:33,988 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/f3cd99ee2f344844be447bebc0574c04 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/f3cd99ee2f344844be447bebc0574c04 2023-03-20 00:57:33,991 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/f3cd99ee2f344844be447bebc0574c04, entries=150, sequenceid=267, filesize=11.9 K 2023-03-20 00:57:33,992 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/6bd281c0f57045a6ba17985bba36178d as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/6bd281c0f57045a6ba17985bba36178d 2023-03-20 00:57:33,998 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/6bd281c0f57045a6ba17985bba36178d, entries=150, sequenceid=267, filesize=11.9 K 2023-03-20 00:57:33,999 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 718cd5d8d8fa4a8a2d92da64e6fb813f in 489ms, sequenceid=267, compaction requested=false 2023-03-20 00:57:33,999 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:33,999 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:33,999 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:33,999 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:33,999 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:33,999 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:33,999 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:34,002 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,002 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:34,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:34,003 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,003 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:34,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:34,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:34,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:34,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:34,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:34,004 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:34,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@822b6a4[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:34,005 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,005 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,005 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:34,005 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:34,005 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:34,005 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,005 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:34,006 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,006 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:34,006 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:34,006 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:34,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:34,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:34,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:34,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:34,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:34,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:34,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:34,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:34,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:34,008 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:34,009 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:34,009 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:34,009 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:34,009 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:34,010 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,010 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,010 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:34,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:34,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:34,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:34,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:34,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:34,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,012 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:34,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:34,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:34,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,014 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:34,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:34,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,024 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:34,024 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:34,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:34,024 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:34,024 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:34,024 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,024 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,024 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,024 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:34,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:34,024 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:34,024 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:34,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:34,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:34,024 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:34,024 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:34,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:34,101 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 718cd5d8d8fa4a8a2d92da64e6fb813f 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-20 00:57:34,102 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=A 2023-03-20 00:57:34,102 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:34,102 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=B 2023-03-20 00:57:34,102 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:34,102 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=C 2023-03-20 00:57:34,102 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:34,110 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/12ce6edebe224ca993a9ae8bb00fd1af 2023-03-20 00:57:34,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:34,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52358 deadline: 1679273914110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:34,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52302 deadline: 1679273914111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:34,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273914116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:34,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273914118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:34,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273914118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,124 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:34,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:34,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:34,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:34,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:34,142 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:34,142 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:34,142 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:34,142 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:34,142 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,142 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:34,143 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:34,143 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:34,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:34,144 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:34,144 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:34,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:34,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:34,144 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:34,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:34,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:34,145 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:34,145 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:34,145 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:34,145 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:34,145 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:34,145 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:34,145 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:34,145 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:34,145 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:34,145 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:34,145 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:34,145 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,145 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:34,145 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:34,146 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:34,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:34,146 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:34,147 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,147 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:34,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:34,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@242e3633[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:34,151 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:34,151 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,152 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,152 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:34,152 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:34,152 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:34,153 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:34,153 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,153 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:34,153 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:34,153 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:34,153 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:34,153 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:34,153 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:34,153 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:34,153 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:34,153 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:34,155 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,155 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,155 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:34,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:34,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:34,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:34,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:34,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:34,157 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:34,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:34,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:34,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,158 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:34,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:34,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,160 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:34,160 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:34,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:34,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:34,160 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,160 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:34,160 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:34,160 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:34,160 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:34,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:34,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:34,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:34,160 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,160 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,161 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,161 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:34,161 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:34,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:34,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52358 deadline: 1679273914219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:34,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52302 deadline: 1679273914220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:34,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273914221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:34,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273914222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:34,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273914222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,261 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:34,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:34,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:34,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:34,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:34,263 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:34,263 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:34,263 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:34,263 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:34,263 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,263 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:34,264 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:34,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:34,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:34,270 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:34,270 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,270 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:34,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:34,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:34,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:34,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:34,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:34,271 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:34,271 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:34,271 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:34,277 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:34,293 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:34,294 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:34,294 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:34,294 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:34,294 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:34,294 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:34,294 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:34,294 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,294 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:34,294 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:34,296 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,296 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:34,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:34,297 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:34,297 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:34,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:34,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:34,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:34,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:34,298 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:34,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b619740[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:34,298 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,299 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,299 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:34,299 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:34,299 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:34,300 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,300 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:34,303 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,303 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:34,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:34,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:34,303 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:34,303 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:34,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:34,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:34,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:34,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:34,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:34,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:34,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:34,305 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:34,305 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:34,305 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:34,305 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:34,305 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:34,316 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,316 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:34,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,316 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:34,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:34,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:34,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:34,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:34,318 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:34,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:34,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:34,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,319 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:34,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:34,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,324 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:34,324 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,324 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:34,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:34,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:34,324 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:34,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:34,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:34,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:34,324 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:34,324 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:34,325 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:34,325 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:34,325 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,325 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,325 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,325 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:34,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:34,425 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:34,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:34,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:34,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:34,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:34,429 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:34,429 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:34,429 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:34,429 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:34,429 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,429 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:34,430 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:34,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:34,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:34,430 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:34,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,431 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,431 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:34,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:34,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:34,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:34,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:34,432 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:34,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:34,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52358 deadline: 1679273914423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,432 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:34,432 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:34,432 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:34,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:34,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273914428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:34,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273914428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:34,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52302 deadline: 1679273914428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:34,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273914429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,437 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:34,437 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:34,437 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:34,437 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:34,437 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:34,437 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:34,437 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:34,437 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:34,437 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,437 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:34,437 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:34,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,441 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:34,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:34,442 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,442 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:34,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:34,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:34,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:34,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:34,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:34,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1309ce62[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:34,443 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:34,443 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,447 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,448 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:34,448 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:34,448 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:34,448 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,448 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:34,449 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:34,449 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:34,449 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:34,449 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:34,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:34,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:34,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:34,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:34,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:34,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:34,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:34,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:34,451 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:34,452 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:34,452 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:34,452 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:34,452 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:34,453 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,453 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,453 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:34,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:34,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:34,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:34,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:34,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,454 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:34,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:34,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:34,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:34,457 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:34,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:34,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,464 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,464 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:34,464 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:34,464 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:34,464 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:34,464 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:34,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:34,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:34,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:34,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:34,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:34,464 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,465 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,465 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:34,465 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,465 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:34,465 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:34,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:34,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:34,532 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/6aba4217782143399a76a7c7c5e4635a 2023-03-20 00:57:34,548 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/6f45ad2a8672434fa7221424e91892c2 2023-03-20 00:57:34,556 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/12ce6edebe224ca993a9ae8bb00fd1af as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/12ce6edebe224ca993a9ae8bb00fd1af 2023-03-20 00:57:34,563 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/12ce6edebe224ca993a9ae8bb00fd1af, entries=150, sequenceid=290, filesize=11.9 K 2023-03-20 00:57:34,563 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/6aba4217782143399a76a7c7c5e4635a as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/6aba4217782143399a76a7c7c5e4635a 2023-03-20 00:57:34,565 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:34,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:34,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:34,573 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/6aba4217782143399a76a7c7c5e4635a, entries=150, sequenceid=290, filesize=11.9 K 2023-03-20 00:57:34,574 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/6f45ad2a8672434fa7221424e91892c2 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/6f45ad2a8672434fa7221424e91892c2 2023-03-20 00:57:34,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:34,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:34,574 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:34,574 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:34,574 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:34,574 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:34,574 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,575 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:34,575 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:34,575 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:34,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:34,575 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,576 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:34,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:34,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:34,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:34,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:34,576 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:34,580 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/6f45ad2a8672434fa7221424e91892c2, entries=150, sequenceid=290, filesize=11.9 K 2023-03-20 00:57:34,580 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 718cd5d8d8fa4a8a2d92da64e6fb813f in 479ms, sequenceid=290, compaction requested=true 2023-03-20 00:57:34,581 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:34,581 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:34,581 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:57:34,581 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:57:34,581 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:34,581 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:34,582 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36818 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:57:34,582 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 718cd5d8d8fa4a8a2d92da64e6fb813f/A is initiating minor compaction (all files) 2023-03-20 00:57:34,582 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 718cd5d8d8fa4a8a2d92da64e6fb813f/A in TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:34,582 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/a4d62aa425a34b8d8498a4491da9bf6a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/44418d2305964c32b1ec416c15ebaf18, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/12ce6edebe224ca993a9ae8bb00fd1af] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp, totalSize=36.0 K 2023-03-20 00:57:34,583 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:34,583 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:34,583 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:34,583 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:34,583 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:34,583 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:34,583 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:34,583 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:34,583 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:34,583 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 718cd5d8d8fa4a8a2d92da64e6fb813f 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-20 00:57:34,583 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=A 2023-03-20 00:57:34,584 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36818 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:57:34,584 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:34,584 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 718cd5d8d8fa4a8a2d92da64e6fb813f/B is initiating minor compaction (all files) 2023-03-20 00:57:34,584 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=B 2023-03-20 00:57:34,584 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 718cd5d8d8fa4a8a2d92da64e6fb813f/B in TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:34,584 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:34,584 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting a4d62aa425a34b8d8498a4491da9bf6a, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1679273851929 2023-03-20 00:57:34,584 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/7759bbe5d17647a5974f880c3465954e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/f3cd99ee2f344844be447bebc0574c04, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/6aba4217782143399a76a7c7c5e4635a] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp, totalSize=36.0 K 2023-03-20 00:57:34,585 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=C 2023-03-20 00:57:34,585 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:34,586 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 44418d2305964c32b1ec416c15ebaf18, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1679273852611 2023-03-20 00:57:34,586 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 7759bbe5d17647a5974f880c3465954e, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1679273851929 2023-03-20 00:57:34,586 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting f3cd99ee2f344844be447bebc0574c04, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1679273852611 2023-03-20 00:57:34,587 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 12ce6edebe224ca993a9ae8bb00fd1af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1679273853778 2023-03-20 00:57:34,587 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 6aba4217782143399a76a7c7c5e4635a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1679273853778 2023-03-20 00:57:34,595 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 718cd5d8d8fa4a8a2d92da64e6fb813f#A#compaction#397 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:34,599 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 718cd5d8d8fa4a8a2d92da64e6fb813f#B#compaction#398 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:34,631 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/319426495254491a8431e02909952f2a as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/319426495254491a8431e02909952f2a 2023-03-20 00:57:34,637 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 718cd5d8d8fa4a8a2d92da64e6fb813f/A of 718cd5d8d8fa4a8a2d92da64e6fb813f into 319426495254491a8431e02909952f2a(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:34,637 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:34,637 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f., storeName=718cd5d8d8fa4a8a2d92da64e6fb813f/A, priority=13, startTime=1679273854581; duration=0sec 2023-03-20 00:57:34,637 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:34,637 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:57:34,638 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36818 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:57:34,638 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 718cd5d8d8fa4a8a2d92da64e6fb813f/C is initiating minor compaction (all files) 2023-03-20 00:57:34,638 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 718cd5d8d8fa4a8a2d92da64e6fb813f/C in TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:34,638 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/1f9704b490b64e90b96655c22b7d5cd6, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/6bd281c0f57045a6ba17985bba36178d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/6f45ad2a8672434fa7221424e91892c2] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp, totalSize=36.0 K 2023-03-20 00:57:34,639 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 1f9704b490b64e90b96655c22b7d5cd6, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1679273851929 2023-03-20 00:57:34,639 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 6bd281c0f57045a6ba17985bba36178d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1679273852611 2023-03-20 00:57:34,640 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 6f45ad2a8672434fa7221424e91892c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1679273853778 2023-03-20 00:57:34,651 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 718cd5d8d8fa4a8a2d92da64e6fb813f#C#compaction#399 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:34,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:34,737 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:34,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679273854747, max timestamp=1679273854765Num uniques 859; 2023-03-20 00:57:34,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-20 00:57:34,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679273854747, max timestamp=1679273854765Num uniques 859; 2023-03-20 00:57:34,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-20 00:57:34,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679273854747, max timestamp=1679273854765Num uniques 859; 2023-03-20 00:57:34,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-20 00:57:34,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:34,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273914780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:34,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273914786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:34,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52302 deadline: 1679273914787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,796 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:34,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:34,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52358 deadline: 1679273914787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273914789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,895 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:34,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273914889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:34,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273914897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:34,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273914897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:34,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52358 deadline: 1679273914897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:34,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:34,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52302 deadline: 1679273914899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,019 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/473d1f8c260f4ff2980ae97c28592c42 2023-03-20 00:57:35,030 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/726ca955572b453e9b0545c1fbd32007 2023-03-20 00:57:35,035 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/f5f4622fee0b4075acc91ac9dd0c1daf as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/f5f4622fee0b4075acc91ac9dd0c1daf 2023-03-20 00:57:35,047 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 718cd5d8d8fa4a8a2d92da64e6fb813f/B of 718cd5d8d8fa4a8a2d92da64e6fb813f into f5f4622fee0b4075acc91ac9dd0c1daf(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:35,047 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:35,047 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f., storeName=718cd5d8d8fa4a8a2d92da64e6fb813f/B, priority=13, startTime=1679273854581; duration=0sec 2023-03-20 00:57:35,048 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:35,067 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/2197f4a13d7e4508ac39fb1ffc104cfd as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/2197f4a13d7e4508ac39fb1ffc104cfd 2023-03-20 00:57:35,082 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 718cd5d8d8fa4a8a2d92da64e6fb813f/C of 718cd5d8d8fa4a8a2d92da64e6fb813f into 2197f4a13d7e4508ac39fb1ffc104cfd(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:35,082 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:35,082 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f., storeName=718cd5d8d8fa4a8a2d92da64e6fb813f/C, priority=13, startTime=1679273854581; duration=0sec 2023-03-20 00:57:35,083 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:35,099 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:35,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273915097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,107 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:35,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273915104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:35,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273915104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:35,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52358 deadline: 1679273915104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:35,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52302 deadline: 1679273915106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:35,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273915400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,410 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:35,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273915409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,410 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:35,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52358 deadline: 1679273915409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,415 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:35,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273915410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:35,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52302 deadline: 1679273915415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,454 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/80d95068393348089fd6893e42c13baa 2023-03-20 00:57:35,458 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/473d1f8c260f4ff2980ae97c28592c42 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/473d1f8c260f4ff2980ae97c28592c42 2023-03-20 00:57:35,461 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/473d1f8c260f4ff2980ae97c28592c42, entries=150, sequenceid=302, filesize=11.9 K 2023-03-20 00:57:35,462 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/726ca955572b453e9b0545c1fbd32007 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/726ca955572b453e9b0545c1fbd32007 2023-03-20 00:57:35,467 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/726ca955572b453e9b0545c1fbd32007, entries=150, sequenceid=302, filesize=11.9 K 2023-03-20 00:57:35,467 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/80d95068393348089fd6893e42c13baa as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/80d95068393348089fd6893e42c13baa 2023-03-20 00:57:35,471 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/80d95068393348089fd6893e42c13baa, entries=150, sequenceid=302, filesize=11.9 K 2023-03-20 00:57:35,472 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for 718cd5d8d8fa4a8a2d92da64e6fb813f in 889ms, sequenceid=302, compaction requested=false 2023-03-20 00:57:35,472 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:35,472 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:35,472 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:35,472 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:35,472 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,472 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:35,472 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:35,473 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,473 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:35,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:35,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:35,474 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:35,474 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:35,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:35,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:35,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:35,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:35,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:35,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e03dd31[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:35,475 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:35,475 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:35,476 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:35,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:35,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:35,476 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:35,476 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:35,476 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,476 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:35,476 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:35,477 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:35,477 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:35,477 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,477 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:35,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:35,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:35,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:35,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:35,479 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:35,479 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:35,479 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:35,479 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:35,479 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:35,480 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:35,480 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:35,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:35,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:35,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:35,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:35,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:35,480 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:35,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:35,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:35,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:35,481 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:35,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:35,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:35,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:35,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,482 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:35,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:35,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,491 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:35,491 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:35,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:35,491 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:35,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:35,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:35,491 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:35,491 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:35,493 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:35,493 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:35,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:35,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:35,493 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:35,493 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,494 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:35,494 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:35,494 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:35,594 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:35,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:35,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:35,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:35,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:35,597 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:35,597 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:35,597 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:35,597 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:35,597 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:35,597 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:35,598 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:35,598 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:35,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:35,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:35,598 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,598 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:35,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:35,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:35,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:35,599 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:35,599 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:35,599 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:35,599 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:35,599 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:35,599 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:35,599 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:35,599 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:35,599 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:35,600 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:35,600 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 718cd5d8d8fa4a8a2d92da64e6fb813f 3/3 column families, dataSize=127.47 KB heapSize=346.31 KB 2023-03-20 00:57:35,600 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=A 2023-03-20 00:57:35,600 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:57:35,600 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=B 2023-03-20 00:57:35,600 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:57:35,600 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=C 2023-03-20 00:57:35,600 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:57:35,610 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/18b8de7b98404aae87dd7eb03ddc29a8 2023-03-20 00:57:35,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:35,909 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:35,928 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:35,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52302 deadline: 1679273915921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:35,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273915923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:35,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273915923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:35,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273915927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:35,935 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:35,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52358 deadline: 1679273915928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,028 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/4a95275e27d04cb0abf994f722c96c82 2023-03-20 00:57:36,036 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:36,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273916030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,036 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:36,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273916031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:36,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273916033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:36,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52358 deadline: 1679273916036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:36,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273916237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:36,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273916238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:36,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273916239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,244 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:36,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52358 deadline: 1679273916240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,445 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/b5a7ebe6381f4f1ba48bd6d9f9a348d2 2023-03-20 00:57:36,450 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/18b8de7b98404aae87dd7eb03ddc29a8 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/18b8de7b98404aae87dd7eb03ddc29a8 2023-03-20 00:57:36,454 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/18b8de7b98404aae87dd7eb03ddc29a8, entries=150, sequenceid=337, filesize=11.9 K 2023-03-20 00:57:36,455 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/4a95275e27d04cb0abf994f722c96c82 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/4a95275e27d04cb0abf994f722c96c82 2023-03-20 00:57:36,458 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/4a95275e27d04cb0abf994f722c96c82, entries=150, sequenceid=337, filesize=11.9 K 2023-03-20 00:57:36,459 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/b5a7ebe6381f4f1ba48bd6d9f9a348d2 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/b5a7ebe6381f4f1ba48bd6d9f9a348d2 2023-03-20 00:57:36,466 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/b5a7ebe6381f4f1ba48bd6d9f9a348d2, entries=150, sequenceid=337, filesize=11.9 K 2023-03-20 00:57:36,467 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~346.27 KB/354576, currentSize=60.38 KB/61830 for 718cd5d8d8fa4a8a2d92da64e6fb813f in 866ms, sequenceid=337, compaction requested=true 2023-03-20 00:57:36,467 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:36,467 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:36,467 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:36,467 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:36,467 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,467 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:36,467 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:36,468 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,468 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:36,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:36,468 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,468 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:36,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:36,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:36,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:36,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:36,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:36,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bc9ed6c[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:36,470 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:36,470 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,470 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,470 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,471 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:36,470 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:36,471 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:36,471 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:36,471 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,471 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:36,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:36,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:36,471 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:36,472 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:36,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:36,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:36,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:36,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:36,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:36,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:36,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:36,475 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:36,475 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:36,475 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:36,475 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:36,475 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:36,483 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,483 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:36,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:36,483 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:36,483 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:36,484 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:36,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:36,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:36,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,491 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:36,491 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:36,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:36,491 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:36,491 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:36,491 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:36,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:36,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:36,491 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:36,491 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:36,491 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,491 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,491 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:36,491 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:36,492 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:36,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:36,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:36,547 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 718cd5d8d8fa4a8a2d92da64e6fb813f 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-20 00:57:36,547 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=A 2023-03-20 00:57:36,547 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:36,547 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=B 2023-03-20 00:57:36,547 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:36,547 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=C 2023-03-20 00:57:36,547 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:36,559 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/ffceb903629d464daf6062282ab23066 2023-03-20 00:57:36,580 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/8dcf1b6e5a244310a715cc31d05b4e15 2023-03-20 00:57:36,590 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:36,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273916579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,590 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:36,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273916580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,590 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:36,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52358 deadline: 1679273916582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:36,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273916583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,592 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:36,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:36,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:36,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:36,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:36,602 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:36,602 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:36,602 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:36,602 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:36,602 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,602 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:36,603 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:36,603 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:36,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:36,603 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,603 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:36,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:36,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:36,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:36,604 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:36,606 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:36,606 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:36,606 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:36,606 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:36,606 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:36,610 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:36,610 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:36,610 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:36,610 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:36,610 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:36,610 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:36,610 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:36,610 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:36,610 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,610 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:36,610 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:36,616 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,616 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:36,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:36,617 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,617 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:36,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:36,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:36,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:36,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:36,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:36,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cf5aac2[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:36,618 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:36,618 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,618 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,618 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:36,619 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:36,619 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:36,619 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,619 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:36,619 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:36,619 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:36,619 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,619 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:36,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:36,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:36,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:36,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:36,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:36,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:36,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:36,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:36,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:36,623 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:36,623 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:36,623 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:36,623 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:36,623 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:36,631 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,631 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,631 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:36,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:36,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:36,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:36,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:36,632 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:36,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:36,633 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:36,634 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,634 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:36,634 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:36,634 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,634 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,634 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,634 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,634 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:36,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:36,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:36,635 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:36,635 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:36,635 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:36,635 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:36,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:36,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:36,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:36,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:36,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:36,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:36,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:36,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273916691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:36,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273916691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:36,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273916692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:36,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52358 deadline: 1679273916692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,735 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:36,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:36,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:36,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:36,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:36,743 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:36,743 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:36,743 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:36,743 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:36,744 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,744 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:36,745 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:36,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:36,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:36,745 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,745 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,745 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:36,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:36,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:36,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:36,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:36,746 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:36,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:36,746 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:36,746 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:36,746 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:36,746 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:36,746 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:36,746 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:36,746 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:36,747 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:36,747 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:36,747 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:36,747 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:36,747 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,747 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:36,747 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:36,747 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,747 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:36,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:36,748 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,748 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:36,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:36,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:36,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:36,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:36,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:36,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@615219e0[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:36,749 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:36,749 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,750 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,750 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:36,750 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:36,750 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:36,750 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,750 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:36,751 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:36,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:36,751 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:36,751 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:36,751 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:36,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:36,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:36,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:36,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:36,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:36,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:36,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:36,753 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:36,753 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:36,753 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:36,753 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:36,753 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:36,754 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,754 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:36,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:36,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:36,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:36,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:36,754 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:36,754 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:36,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:36,756 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,757 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:36,757 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:36,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:36,757 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:36,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:36,757 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:36,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:36,758 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:36,758 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:36,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:36,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:36,758 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,758 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:36,758 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,758 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,758 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,758 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:36,858 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:36,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:36,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:36,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:36,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:36,862 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:36,862 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:36,862 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:36,862 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:36,862 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,862 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:36,863 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,863 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:36,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:36,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:36,863 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,863 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:36,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:36,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:36,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:36,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:36,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:36,864 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:36,864 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:36,864 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:36,866 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:36,866 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:36,866 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:36,866 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:36,866 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:36,866 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:36,866 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:36,866 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:36,866 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,866 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:36,866 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:36,867 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,867 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:36,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:36,867 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,867 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:36,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:36,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:36,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:36,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:36,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:36,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71a8aaae[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:36,868 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:36,868 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,869 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,869 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:36,869 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:36,869 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,869 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:36,869 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:36,869 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:36,869 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,869 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:36,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:36,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:36,870 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:36,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:36,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:36,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:36,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:36,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:36,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:36,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:36,871 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:36,871 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:36,871 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:36,871 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:36,871 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:36,872 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,872 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:36,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:36,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,872 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:36,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:36,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:36,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:36,873 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:36,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:36,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:36,874 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:36,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:36,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,876 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,876 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:36,876 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:36,876 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:36,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:36,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:36,876 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,876 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,876 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,876 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:36,877 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:36,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:36,876 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:36,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:36,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:36,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:36,877 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:36,877 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:36,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:36,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273916895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:36,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273916895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:36,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273916895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:36,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52358 deadline: 1679273916896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,936 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:36,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52302 deadline: 1679273916931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,977 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:36,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:36,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:36,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:36,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:36,980 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:36,980 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:36,980 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:36,980 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:36,980 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,980 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:36,980 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,980 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:36,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:36,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:36,981 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,981 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:36,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:36,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:36,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:36,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:36,981 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:36,981 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:36,981 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:36,981 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:36,981 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:36,981 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:36,982 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:36,982 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:36,982 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:36,982 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:36,982 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:36,982 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:36,982 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:36,982 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:36,982 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,982 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:36,982 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:36,982 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,983 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:36,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:36,983 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:36,983 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:36,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:36,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:36,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:36,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:36,984 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:36,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@651e8b7a[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:36,984 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,986 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:36,986 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:36,990 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:36,990 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:36,990 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:36,990 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:36,995 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/08f9e5e956c24ca6b26aa48761830dc5 2023-03-20 00:57:36,998 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/ffceb903629d464daf6062282ab23066 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/ffceb903629d464daf6062282ab23066 2023-03-20 00:57:37,001 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/ffceb903629d464daf6062282ab23066, entries=200, sequenceid=350, filesize=14.3 K 2023-03-20 00:57:37,002 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/8dcf1b6e5a244310a715cc31d05b4e15 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/8dcf1b6e5a244310a715cc31d05b4e15 2023-03-20 00:57:37,007 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/8dcf1b6e5a244310a715cc31d05b4e15, entries=150, sequenceid=350, filesize=11.9 K 2023-03-20 00:57:37,007 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/08f9e5e956c24ca6b26aa48761830dc5 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/08f9e5e956c24ca6b26aa48761830dc5 2023-03-20 00:57:37,008 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:37,008 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:37,008 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:37,010 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:37,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:37,013 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/08f9e5e956c24ca6b26aa48761830dc5, entries=150, sequenceid=350, filesize=11.9 K 2023-03-20 00:57:37,013 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-03-20 00:57:37,014 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 718cd5d8d8fa4a8a2d92da64e6fb813f in 467ms, sequenceid=350, compaction requested=true 2023-03-20 00:57:37,014 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:37,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:37,014 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 00:57:37,014 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:37,014 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 00:57:37,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:37,014 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:37,014 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:37,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:37,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:37,015 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51696 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 00:57:37,015 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 00:57:37,015 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 718cd5d8d8fa4a8a2d92da64e6fb813f/A is initiating minor compaction (all files) 2023-03-20 00:57:37,015 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 718cd5d8d8fa4a8a2d92da64e6fb813f/B is initiating minor compaction (all files) 2023-03-20 00:57:37,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:37,015 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 718cd5d8d8fa4a8a2d92da64e6fb813f/B in TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:37,015 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 718cd5d8d8fa4a8a2d92da64e6fb813f/A in TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:37,015 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/f5f4622fee0b4075acc91ac9dd0c1daf, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/726ca955572b453e9b0545c1fbd32007, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/4a95275e27d04cb0abf994f722c96c82, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/8dcf1b6e5a244310a715cc31d05b4e15] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp, totalSize=48.1 K 2023-03-20 00:57:37,015 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/319426495254491a8431e02909952f2a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/473d1f8c260f4ff2980ae97c28592c42, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/18b8de7b98404aae87dd7eb03ddc29a8, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/ffceb903629d464daf6062282ab23066] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp, totalSize=50.5 K 2023-03-20 00:57:37,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,016 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting f5f4622fee0b4075acc91ac9dd0c1daf, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1679273853778 2023-03-20 00:57:37,016 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 319426495254491a8431e02909952f2a, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1679273853778 2023-03-20 00:57:37,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:37,016 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 726ca955572b453e9b0545c1fbd32007, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1679273854115 2023-03-20 00:57:37,016 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:37,016 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:37,016 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:37,016 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 473d1f8c260f4ff2980ae97c28592c42, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1679273854115 2023-03-20 00:57:37,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:37,016 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:37,016 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:37,016 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 4a95275e27d04cb0abf994f722c96c82, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1679273854783 2023-03-20 00:57:37,016 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 18b8de7b98404aae87dd7eb03ddc29a8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1679273854783 2023-03-20 00:57:37,016 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 8dcf1b6e5a244310a715cc31d05b4e15, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1679273855926 2023-03-20 00:57:37,017 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting ffceb903629d464daf6062282ab23066, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1679273855922 2023-03-20 00:57:37,017 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:37,017 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:37,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:37,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:37,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:37,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:37,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:37,017 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:37,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:37,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:37,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:37,018 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:37,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:37,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:37,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:37,019 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:37,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:37,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,022 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:37,022 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:37,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:37,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:37,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:37,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:37,023 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 718cd5d8d8fa4a8a2d92da64e6fb813f#B#compaction#408 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:37,023 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,023 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:37,023 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:37,023 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,023 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:37,023 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:37,023 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:37,023 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:37,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:37,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:37,024 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:37,024 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:37,025 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 718cd5d8d8fa4a8a2d92da64e6fb813f#A#compaction#409 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:37,124 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:37,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:37,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:37,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:37,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:37,126 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:37,126 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:37,126 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:37,127 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:37,127 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:37,127 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:37,127 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,128 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:37,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:37,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:37,128 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,128 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:37,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:37,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:37,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:37,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:37,128 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:37,128 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:37,128 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:37,128 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:37,129 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:37,129 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:37,129 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:37,129 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:37,129 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:37,129 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:37,129 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 718cd5d8d8fa4a8a2d92da64e6fb813f 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-20 00:57:37,129 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=A 2023-03-20 00:57:37,129 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:37,129 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=B 2023-03-20 00:57:37,129 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:37,129 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=C 2023-03-20 00:57:37,129 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:37,134 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=369 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/945741f141c44991bbffbab870ff843f 2023-03-20 00:57:37,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:37,207 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:37,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:37,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273917220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:37,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52358 deadline: 1679273917221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:37,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273917223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:37,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273917224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:37,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273917324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:37,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52358 deadline: 1679273917325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:37,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273917332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:37,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273917333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,453 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/59fd2302c6524eeea2c618dfa6bdc797 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/59fd2302c6524eeea2c618dfa6bdc797 2023-03-20 00:57:37,456 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/49d8e83ac383469ba96d842f84ab06ac as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/49d8e83ac383469ba96d842f84ab06ac 2023-03-20 00:57:37,465 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 718cd5d8d8fa4a8a2d92da64e6fb813f/B of 718cd5d8d8fa4a8a2d92da64e6fb813f into 59fd2302c6524eeea2c618dfa6bdc797(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:37,465 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:37,465 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 718cd5d8d8fa4a8a2d92da64e6fb813f/A of 718cd5d8d8fa4a8a2d92da64e6fb813f into 49d8e83ac383469ba96d842f84ab06ac(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:37,465 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f., storeName=718cd5d8d8fa4a8a2d92da64e6fb813f/B, priority=12, startTime=1679273857014; duration=0sec 2023-03-20 00:57:37,465 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:37,465 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f., storeName=718cd5d8d8fa4a8a2d92da64e6fb813f/A, priority=12, startTime=1679273857014; duration=0sec 2023-03-20 00:57:37,465 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:37,465 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:37,465 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 00:57:37,466 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 00:57:37,466 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 718cd5d8d8fa4a8a2d92da64e6fb813f/C is initiating minor compaction (all files) 2023-03-20 00:57:37,466 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 718cd5d8d8fa4a8a2d92da64e6fb813f/C in TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:37,467 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/2197f4a13d7e4508ac39fb1ffc104cfd, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/80d95068393348089fd6893e42c13baa, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/b5a7ebe6381f4f1ba48bd6d9f9a348d2, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/08f9e5e956c24ca6b26aa48761830dc5] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp, totalSize=48.1 K 2023-03-20 00:57:37,467 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 2197f4a13d7e4508ac39fb1ffc104cfd, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1679273853778 2023-03-20 00:57:37,468 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 80d95068393348089fd6893e42c13baa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1679273854115 2023-03-20 00:57:37,469 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting b5a7ebe6381f4f1ba48bd6d9f9a348d2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1679273854783 2023-03-20 00:57:37,470 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 08f9e5e956c24ca6b26aa48761830dc5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1679273855926 2023-03-20 00:57:37,484 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 718cd5d8d8fa4a8a2d92da64e6fb813f#C#compaction#412 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:37,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:37,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273917529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:37,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52358 deadline: 1679273917529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,542 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=369 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/0c490aae7ec949299bb281fc08f3041f 2023-03-20 00:57:37,543 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:37,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273917538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,543 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:37,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273917538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,561 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=369 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/52db05dd9c9b43e99e2aa540ad7bb44c 2023-03-20 00:57:37,564 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/945741f141c44991bbffbab870ff843f as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/945741f141c44991bbffbab870ff843f 2023-03-20 00:57:37,567 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/945741f141c44991bbffbab870ff843f, entries=150, sequenceid=369, filesize=11.9 K 2023-03-20 00:57:37,568 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/0c490aae7ec949299bb281fc08f3041f as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/0c490aae7ec949299bb281fc08f3041f 2023-03-20 00:57:37,571 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/0c490aae7ec949299bb281fc08f3041f, entries=150, sequenceid=369, filesize=11.9 K 2023-03-20 00:57:37,573 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/52db05dd9c9b43e99e2aa540ad7bb44c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/52db05dd9c9b43e99e2aa540ad7bb44c 2023-03-20 00:57:37,576 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/52db05dd9c9b43e99e2aa540ad7bb44c, entries=150, sequenceid=369, filesize=11.9 K 2023-03-20 00:57:37,577 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 718cd5d8d8fa4a8a2d92da64e6fb813f in 447ms, sequenceid=369, compaction requested=false 2023-03-20 00:57:37,577 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:37,577 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:37,577 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:37,577 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:37,577 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,577 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:37,577 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:37,578 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:37,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:37,578 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:37,578 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:37,578 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:37,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:37,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:37,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:37,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:37,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:37,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74ac9ea1[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:37,579 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:37,580 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:37,580 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:37,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:37,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:37,580 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:37,580 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:37,580 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,580 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:37,580 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:37,581 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,581 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:37,581 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:37,581 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:37,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:37,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:37,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:37,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:37,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:37,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:37,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:37,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:37,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:37,583 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:37,583 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:37,583 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:37,584 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:37,584 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:37,584 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:37,584 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:37,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:37,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:37,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:37,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:37,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:37,584 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:37,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:37,585 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:37,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:37,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:37,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:37,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:37,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:37,586 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:37,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:37,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,588 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:37,588 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:37,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:37,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:37,588 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,588 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:37,588 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:37,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:37,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:37,588 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:37,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:37,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:37,589 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,589 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:37,589 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:37,589 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:37,589 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:37,589 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:37,689 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:37,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:37,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:37,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:37,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:37,691 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:37,691 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:37,691 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:37,691 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:37,692 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:37,692 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:37,692 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:37,692 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:37,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:37,693 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,693 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:37,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:37,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:37,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:37,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:37,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:37,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:37,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:37,696 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:37,696 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:37,696 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:37,696 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:37,696 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:37,696 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:37,696 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:37,696 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 718cd5d8d8fa4a8a2d92da64e6fb813f 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-20 00:57:37,696 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=A 2023-03-20 00:57:37,696 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:37,697 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=B 2023-03-20 00:57:37,697 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:37,697 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=C 2023-03-20 00:57:37,697 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:37,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:37,837 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:37,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:37,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273917857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:37,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273917857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:37,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52358 deadline: 1679273917862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:37,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273917863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,900 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/2c9fdf0fdf0b43ca8882a62553804d29 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/2c9fdf0fdf0b43ca8882a62553804d29 2023-03-20 00:57:37,903 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 718cd5d8d8fa4a8a2d92da64e6fb813f/C of 718cd5d8d8fa4a8a2d92da64e6fb813f into 2c9fdf0fdf0b43ca8882a62553804d29(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:37,904 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:37,904 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f., storeName=718cd5d8d8fa4a8a2d92da64e6fb813f/C, priority=12, startTime=1679273857014; duration=0sec 2023-03-20 00:57:37,904 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:37,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:37,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273917963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,970 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:37,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273917963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,970 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:37,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52358 deadline: 1679273917965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:37,970 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:37,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273917965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,102 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/4f0a005216814c298de5d7751e93cbf7 2023-03-20 00:57:38,112 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/20c29cd560634c2489960bd27da83642 2023-03-20 00:57:38,123 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/298129c8a2be439aadf2424535f78ef3 2023-03-20 00:57:38,127 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/4f0a005216814c298de5d7751e93cbf7 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/4f0a005216814c298de5d7751e93cbf7 2023-03-20 00:57:38,132 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/4f0a005216814c298de5d7751e93cbf7, entries=150, sequenceid=384, filesize=11.9 K 2023-03-20 00:57:38,132 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/20c29cd560634c2489960bd27da83642 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/20c29cd560634c2489960bd27da83642 2023-03-20 00:57:38,135 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/20c29cd560634c2489960bd27da83642, entries=150, sequenceid=384, filesize=11.9 K 2023-03-20 00:57:38,136 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/298129c8a2be439aadf2424535f78ef3 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/298129c8a2be439aadf2424535f78ef3 2023-03-20 00:57:38,139 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/298129c8a2be439aadf2424535f78ef3, entries=150, sequenceid=384, filesize=11.9 K 2023-03-20 00:57:38,140 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 718cd5d8d8fa4a8a2d92da64e6fb813f in 444ms, sequenceid=384, compaction requested=true 2023-03-20 00:57:38,140 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:38,140 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:38,140 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:38,140 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:38,140 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,140 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:38,140 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:38,141 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,141 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:38,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:38,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:38,141 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,141 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:38,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:38,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:38,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:38,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c014cd[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:38,142 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:38,142 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,143 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,143 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,143 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:38,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,143 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:38,143 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:38,143 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:38,144 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,144 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:38,144 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:38,144 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:38,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:38,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:38,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:38,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:38,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:38,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:38,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:38,146 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:38,146 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:38,146 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:38,146 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:38,146 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:38,146 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:38,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:38,147 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,147 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:38,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:38,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:38,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:38,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:38,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:38,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,149 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,150 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,151 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,151 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,151 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:38,151 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,151 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:38,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:38,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:38,151 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:38,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:38,151 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:38,151 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:38,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:38,151 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,152 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,151 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:38,152 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:38,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:38,174 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 718cd5d8d8fa4a8a2d92da64e6fb813f 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-20 00:57:38,174 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=A 2023-03-20 00:57:38,174 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:38,174 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=B 2023-03-20 00:57:38,174 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:38,174 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=C 2023-03-20 00:57:38,174 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:38,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:38,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52358 deadline: 1679273918182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,186 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:38,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273918182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,187 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:38,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273918183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:38,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273918185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,252 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:38,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:38,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:38,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:38,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:38,254 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:38,254 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:38,254 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:38,254 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:38,255 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,255 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,256 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:38,256 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:38,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:38,257 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,257 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:38,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:38,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,260 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:38,261 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:38,261 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:38,261 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:38,261 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:38,261 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:38,262 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:38,262 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:38,262 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:38,262 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:38,262 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:38,262 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:38,262 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:38,262 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:38,262 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,262 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:38,262 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:38,267 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,267 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:38,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:38,267 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,267 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:38,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:38,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:38,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:38,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:38,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31092301[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:38,268 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:38,268 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,269 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,269 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:38,269 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:38,269 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:38,269 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,269 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:38,270 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:38,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:38,270 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:38,270 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:38,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:38,270 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:38,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:38,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:38,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:38,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:38,272 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:38,272 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:38,272 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:38,272 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:38,272 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:38,274 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,274 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:38,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:38,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,274 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:38,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:38,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:38,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:38,275 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:38,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,276 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:38,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,278 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:38,278 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,278 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:38,278 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:38,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:38,278 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:38,278 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,278 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,278 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,278 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:38,278 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,278 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,278 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:38,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:38,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:38,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:38,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:38,291 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:38,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52358 deadline: 1679273918286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,291 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:38,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273918287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:38,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273918288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,295 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:38,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273918291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,378 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:38,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:38,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:38,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:38,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:38,381 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:38,381 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:38,381 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:38,381 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:38,381 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,381 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,382 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:38,382 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:38,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:38,382 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,382 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:38,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:38,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:38,383 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:38,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:38,384 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:38,384 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:38,384 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:38,384 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:38,384 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:38,384 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:38,384 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:38,384 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:38,384 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:38,384 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:38,384 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:38,384 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,384 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:38,384 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:38,385 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,385 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:38,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:38,385 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,385 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:38,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:38,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:38,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:38,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:38,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20dd2b01[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:38,386 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:38,386 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,387 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,387 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:38,387 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:38,387 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,387 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:38,387 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:38,388 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,388 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:38,388 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:38,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:38,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:38,388 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:38,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:38,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:38,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:38,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:38,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:38,390 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:38,390 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:38,390 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:38,390 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:38,390 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:38,390 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,390 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:38,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:38,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,390 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:38,391 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:38,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:38,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:38,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:38,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,392 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:38,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,393 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,393 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,393 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,393 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:38,393 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:38,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:38,393 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:38,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:38,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:38,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:38,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:38,393 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,393 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,393 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:38,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:38,393 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,393 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:38,393 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:38,494 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:38,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:38,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:38,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:38,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273918492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:38,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52358 deadline: 1679273918492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,498 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:38,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273918494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:38,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:38,499 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:38,499 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:38,499 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:38,500 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:38,500 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,500 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,501 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:38,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:38,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:38,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,502 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,503 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,503 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:38,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:38,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,503 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:38,504 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:38,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:38,504 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:38,504 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:38,504 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:38,504 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:38,504 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:38,504 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:38,504 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:38,504 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:38,504 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:38,504 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:38,504 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:38,504 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,504 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:38,504 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:38,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:38,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273918499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,514 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:38,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:38,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:38,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:38,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:38,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:38,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3094e02d[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:38,516 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:38,516 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,517 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,517 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:38,517 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,517 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,517 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,517 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:38,517 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:38,517 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:38,517 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:38,518 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:38,518 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:38,518 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:38,519 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:38,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:38,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:38,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:38,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:38,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:38,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:38,525 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:38,525 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:38,525 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:38,525 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:38,525 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:38,533 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,533 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:38,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:38,533 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:38,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:38,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:38,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:38,534 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:38,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,535 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:38,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,543 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,543 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,543 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,543 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,543 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,543 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:38,543 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:38,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:38,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:38,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:38,544 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:38,544 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:38,543 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,543 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:38,543 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,544 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:38,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:38,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:38,581 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/cfa97321cf044535875ae344df115e0b 2023-03-20 00:57:38,644 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:38,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:38,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:38,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:38,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:38,646 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:38,646 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:38,646 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:38,646 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:38,646 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,646 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,647 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,647 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:38,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:38,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:38,647 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,647 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:38,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:38,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,648 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:38,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:38,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:38,648 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:38,648 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:38,648 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:38,648 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:38,648 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:38,648 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:38,648 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:38,648 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:38,649 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:38,649 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:38,649 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:38,649 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,649 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:38,649 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:38,650 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,650 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:38,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:38,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:38,651 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,651 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:38,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:38,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:38,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:38,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3577fc8e[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:38,653 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:38,653 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,654 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,654 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:38,654 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:38,654 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,654 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:38,654 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:38,655 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,655 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:38,655 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:38,655 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:38,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:38,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:38,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:38,657 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:38,657 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:38,657 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:38,657 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:38,657 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:38,658 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,658 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:38,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:38,658 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:38,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:38,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:38,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:38,658 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:38,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,660 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:38,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,662 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,662 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,662 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:38,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:38,662 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:38,662 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:38,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:38,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:38,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:38,662 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,662 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:38,663 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,663 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:38,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:38,663 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:38,663 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:38,663 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,763 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:38,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:38,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:38,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:38,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:38,766 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:38,766 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:38,766 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:38,766 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:38,768 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,768 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,768 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:38,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:38,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:38,769 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,769 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,769 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:38,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:38,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,769 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:38,770 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:38,770 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:38,770 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:38,770 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:38,770 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:38,770 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:38,770 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:38,771 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:38,771 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:38,771 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:38,771 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:38,771 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:38,771 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:38,771 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,771 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:38,771 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:38,772 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,772 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:38,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:38,772 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:38,772 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:38,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:38,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:38,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:38,774 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:38,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d9ed00a[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:38,774 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,776 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,776 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,776 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:38,776 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:38,776 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:38,776 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:38,777 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:38,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:38,777 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:38,777 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:38,777 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:38,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:38,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:38,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:38,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:38,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:38,781 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:38,781 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:38,781 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:38,781 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:38,781 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:38,783 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,783 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:38,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,783 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:38,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:38,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:38,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:38,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,783 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:38,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:38,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,785 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:38,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,787 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:38,787 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:38,788 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,788 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,788 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,788 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,788 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:38,788 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:38,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:38,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:38,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:38,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:38,788 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,789 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,789 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:38,789 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:38,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:38,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:38,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:38,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52358 deadline: 1679273918799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:38,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273918800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,805 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:38,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273918801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:38,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273918810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,889 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:38,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:38,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:38,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:38,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:38,891 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:38,892 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:38,892 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:38,892 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:38,892 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,892 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,892 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,892 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:38,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:38,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:38,893 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,893 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:38,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:38,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,893 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:38,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:38,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:38,894 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:38,894 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:38,894 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:38,894 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:38,894 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:38,894 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:38,894 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:38,894 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:38,894 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:38,894 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:38,894 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:38,894 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,894 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:38,894 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:38,895 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,895 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:38,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:38,895 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,895 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:38,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:38,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:38,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:38,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:38,896 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:38,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d27f9b[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:38,896 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,897 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,897 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,897 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:38,897 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:38,897 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:38,897 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:38,898 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,898 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:38,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:38,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:38,898 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:38,898 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:38,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:38,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:38,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:38,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:38,900 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:38,900 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:38,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:38,900 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:38,900 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:38,900 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:38,901 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,901 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:38,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,901 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:38,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:38,901 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:38,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:38,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:38,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:38,902 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:38,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:38,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,903 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:38,903 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:38,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:38,904 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:38,903 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,904 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:38,904 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,904 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,904 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:38,904 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:38,903 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:38,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:38,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:38,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:38,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:38,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:38,904 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:38,904 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:38,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:38,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52302 deadline: 1679273918947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:38,952 DEBUG [Thread-2564] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4164 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f., hostname=jenkins-hbase20.apache.org,34553,1679273725199, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 00:57:39,004 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:39,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:39,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:39,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:39,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:39,007 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:39,007 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:39,007 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:39,007 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:39,007 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,007 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:39,008 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/7ee586579d834836a4e361bc9a2741ff 2023-03-20 00:57:39,008 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:39,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:39,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:39,008 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:39,009 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,009 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:39,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:39,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:39,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:39,009 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:39,009 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:39,009 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:39,009 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:39,009 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:39,010 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:39,010 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:39,010 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:39,010 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:39,010 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:39,010 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:39,010 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:39,010 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:39,010 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,010 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:39,010 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:39,011 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,011 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:39,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:39,011 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,011 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:39,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:39,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:39,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:39,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:39,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:39,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13d072a4[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:39,013 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:39,013 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,014 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,014 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:39,014 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,014 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:39,014 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:39,014 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:39,015 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:39,015 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:39,016 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:39,016 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:39,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:39,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:39,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:39,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:39,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:39,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:39,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:39,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:39,018 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:39,018 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:39,018 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:39,018 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:39,018 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:39,019 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:39,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:39,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:39,020 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:39,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:39,021 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,021 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:39,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:39,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:39,021 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:39,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:39,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:39,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,023 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:39,023 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:39,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:39,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:39,023 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:39,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:39,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:39,023 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:39,023 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:39,023 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:39,023 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:39,023 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:39,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:39,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:39,023 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,023 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,023 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,023 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,123 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:39,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:39,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:39,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:39,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:39,126 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:39,126 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:39,126 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:39,126 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:39,126 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,127 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:39,127 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,127 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:39,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:39,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:39,127 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,127 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:39,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:39,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:39,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:39,128 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:39,130 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:39,130 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:39,130 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:39,130 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:39,134 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:39,134 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:39,134 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:39,134 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:39,134 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:39,134 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:39,134 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:39,134 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:39,134 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,134 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:39,134 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:39,141 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,142 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,142 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:39,144 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:39,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:39,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:39,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:39,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:39,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:39,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:39,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19496fdc[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:39,151 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:39,151 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,151 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,151 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,151 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:39,151 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:39,152 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:39,152 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:39,152 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:39,152 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:39,152 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:39,153 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:39,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:39,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:39,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:39,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:39,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:39,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:39,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:39,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:39,155 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:39,155 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:39,155 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:39,155 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:39,155 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:39,156 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:39,156 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,156 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:39,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:39,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:39,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:39,157 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:39,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:39,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:39,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:39,158 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:39,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:39,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,160 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,161 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:39,161 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:39,161 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,161 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,161 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,161 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,161 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:39,161 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:39,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:39,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:39,161 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:39,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:39,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:39,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:39,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:39,162 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:39,162 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:39,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:39,262 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:39,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:39,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:39,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:39,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:39,267 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:39,267 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:39,267 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:39,267 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:39,267 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,267 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:39,268 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,268 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:39,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:39,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:39,268 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,268 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:39,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:39,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:39,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:39,269 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:39,282 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:39,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:39,282 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:39,282 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:39,282 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:39,290 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:39,290 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:39,290 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:39,290 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:39,290 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:39,290 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:39,290 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:39,290 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:39,290 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,290 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:39,290 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:39,292 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,292 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:39,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:39,293 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,293 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:39,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:39,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:39,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:39,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:39,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:39,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22ecf27f[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:39,294 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:39,294 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,294 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,295 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:39,295 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:39,295 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:39,295 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,295 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:39,295 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:39,295 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:39,295 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,295 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:39,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:39,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:39,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:39,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:39,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:39,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:39,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:39,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:39,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:39,298 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:39,298 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:39,298 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:39,298 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:39,298 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:39,306 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,306 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:39,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:39,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,306 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:39,306 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:39,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:39,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:39,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:39,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:39,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:39,308 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:39,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:39,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:39,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52350 deadline: 1679273919307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:39,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52358 deadline: 1679273919307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,313 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:39,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52320 deadline: 1679273919307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,316 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,316 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:39,316 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:39,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:39,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:39,316 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:39,316 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:39,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:39,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:39,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:39,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:39,317 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:39,317 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,317 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,317 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,317 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:39,317 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:39,317 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:39,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:39,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52300 deadline: 1679273919314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,417 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:39,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:39,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:39,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:39,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:39,428 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:39,428 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:39,428 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/674e416095cd45bdb8306a31f812d364 2023-03-20 00:57:39,428 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:39,428 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:39,429 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,429 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:39,429 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:39,429 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:39,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:39,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:39,431 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,431 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:39,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:39,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:39,432 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:39,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:39,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:39,432 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:39,432 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:39,432 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:39,432 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:39,432 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:39,432 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:39,433 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:39,433 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:39,433 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:39,433 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/cfa97321cf044535875ae344df115e0b as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/cfa97321cf044535875ae344df115e0b 2023-03-20 00:57:39,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:39,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:39,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:39,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:39,438 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/cfa97321cf044535875ae344df115e0b, entries=200, sequenceid=405, filesize=14.3 K 2023-03-20 00:57:39,439 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/7ee586579d834836a4e361bc9a2741ff as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/7ee586579d834836a4e361bc9a2741ff 2023-03-20 00:57:39,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:39,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:39,441 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,442 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,442 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:39,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:39,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:39,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:39,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:39,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:39,446 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/7ee586579d834836a4e361bc9a2741ff, entries=150, sequenceid=405, filesize=11.9 K 2023-03-20 00:57:39,446 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:39,446 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35d56f1e[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:39,446 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/674e416095cd45bdb8306a31f812d364 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/674e416095cd45bdb8306a31f812d364 2023-03-20 00:57:39,450 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,450 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,450 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:39,450 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:39,450 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:39,450 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:39,451 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/674e416095cd45bdb8306a31f812d364, entries=150, sequenceid=405, filesize=11.9 K 2023-03-20 00:57:39,452 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 718cd5d8d8fa4a8a2d92da64e6fb813f in 1278ms, sequenceid=405, compaction requested=true 2023-03-20 00:57:39,452 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:39,452 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:39,453 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-20 00:57:39,453 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-20 00:57:39,453 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 00:57:39,453 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 00:57:39,456 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51832 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 00:57:39,456 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,456 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 718cd5d8d8fa4a8a2d92da64e6fb813f/A is initiating minor compaction (all files) 2023-03-20 00:57:39,456 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 718cd5d8d8fa4a8a2d92da64e6fb813f/A in TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:39,456 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 00:57:39,456 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/49d8e83ac383469ba96d842f84ab06ac, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/945741f141c44991bbffbab870ff843f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/4f0a005216814c298de5d7751e93cbf7, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/cfa97321cf044535875ae344df115e0b] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp, totalSize=50.6 K 2023-03-20 00:57:39,456 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 718cd5d8d8fa4a8a2d92da64e6fb813f/C is initiating minor compaction (all files) 2023-03-20 00:57:39,456 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 718cd5d8d8fa4a8a2d92da64e6fb813f/C in TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:39,456 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/2c9fdf0fdf0b43ca8882a62553804d29, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/52db05dd9c9b43e99e2aa540ad7bb44c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/298129c8a2be439aadf2424535f78ef3, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/674e416095cd45bdb8306a31f812d364] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp, totalSize=48.2 K 2023-03-20 00:57:39,456 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:39,456 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:39,456 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:39,457 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 49d8e83ac383469ba96d842f84ab06ac, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1679273855926 2023-03-20 00:57:39,457 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 2c9fdf0fdf0b43ca8882a62553804d29, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1679273855926 2023-03-20 00:57:39,457 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 945741f141c44991bbffbab870ff843f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1679273856579 2023-03-20 00:57:39,457 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 52db05dd9c9b43e99e2aa540ad7bb44c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1679273856579 2023-03-20 00:57:39,457 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 4f0a005216814c298de5d7751e93cbf7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1679273857216 2023-03-20 00:57:39,457 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 298129c8a2be439aadf2424535f78ef3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1679273857216 2023-03-20 00:57:39,457 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting cfa97321cf044535875ae344df115e0b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1679273857854 2023-03-20 00:57:39,457 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 674e416095cd45bdb8306a31f812d364, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1679273857861 2023-03-20 00:57:39,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:39,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:39,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:39,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:39,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:39,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:39,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:39,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:39,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:39,463 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:39,463 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:39,463 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:39,463 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:39,463 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:39,467 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,467 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:39,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,469 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 718cd5d8d8fa4a8a2d92da64e6fb813f#C#compaction#420 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:39,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:39,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:39,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,474 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:39,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:39,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:39,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:39,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:39,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:39,475 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:39,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:39,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,476 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,481 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:39,481 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:39,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:39,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:39,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:39,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:39,482 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,482 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:39,482 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:39,482 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:39,482 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:39,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:39,482 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,482 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:39,482 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,482 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:39,482 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:39,491 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 718cd5d8d8fa4a8a2d92da64e6fb813f#A#compaction#421 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:39,582 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:39,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:39,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:39,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:39,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:39,585 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:39,585 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:39,585 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:39,585 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:39,585 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,585 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:39,586 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:39,586 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:39,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:39,586 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:39,586 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:39,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:39,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:39,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:39,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:39,587 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:39,587 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:39,587 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:39,587 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:39,587 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:39,587 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:39,587 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:39,587 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:39,587 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:39,587 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. started... 2023-03-20 00:57:39,587 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 718cd5d8d8fa4a8a2d92da64e6fb813f 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-20 00:57:39,587 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=A 2023-03-20 00:57:39,587 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:39,587 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=B 2023-03-20 00:57:39,587 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:39,588 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=C 2023-03-20 00:57:39,588 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:39,887 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/73aef888c2014d35a262a4a17a6eaa22 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/73aef888c2014d35a262a4a17a6eaa22 2023-03-20 00:57:39,891 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 718cd5d8d8fa4a8a2d92da64e6fb813f/C of 718cd5d8d8fa4a8a2d92da64e6fb813f into 73aef888c2014d35a262a4a17a6eaa22(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:39,891 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:39,891 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f., storeName=718cd5d8d8fa4a8a2d92da64e6fb813f/C, priority=12, startTime=1679273859453; duration=0sec 2023-03-20 00:57:39,891 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:39,891 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 00:57:39,894 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 00:57:39,895 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 718cd5d8d8fa4a8a2d92da64e6fb813f/B is initiating minor compaction (all files) 2023-03-20 00:57:39,895 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 718cd5d8d8fa4a8a2d92da64e6fb813f/B in TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:39,895 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/59fd2302c6524eeea2c618dfa6bdc797, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/0c490aae7ec949299bb281fc08f3041f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/20c29cd560634c2489960bd27da83642, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/7ee586579d834836a4e361bc9a2741ff] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp, totalSize=48.2 K 2023-03-20 00:57:39,895 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 59fd2302c6524eeea2c618dfa6bdc797, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1679273855926 2023-03-20 00:57:39,895 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 0c490aae7ec949299bb281fc08f3041f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1679273856579 2023-03-20 00:57:39,897 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 20c29cd560634c2489960bd27da83642, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1679273857216 2023-03-20 00:57:39,897 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 7ee586579d834836a4e361bc9a2741ff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1679273857861 2023-03-20 00:57:39,908 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 718cd5d8d8fa4a8a2d92da64e6fb813f#B#compaction#423 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:39,914 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/edcf3f9c8eeb48508ae6907414df41a0 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/edcf3f9c8eeb48508ae6907414df41a0 2023-03-20 00:57:39,924 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 718cd5d8d8fa4a8a2d92da64e6fb813f/A of 718cd5d8d8fa4a8a2d92da64e6fb813f into edcf3f9c8eeb48508ae6907414df41a0(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:39,924 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:39,924 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f., storeName=718cd5d8d8fa4a8a2d92da64e6fb813f/A, priority=12, startTime=1679273859452; duration=0sec 2023-03-20 00:57:39,924 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:39,927 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/be5f93b8f885494faf2ddedfe953cef0 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/be5f93b8f885494faf2ddedfe953cef0 2023-03-20 00:57:39,931 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 718cd5d8d8fa4a8a2d92da64e6fb813f/B of 718cd5d8d8fa4a8a2d92da64e6fb813f into be5f93b8f885494faf2ddedfe953cef0(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:39,931 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:39,931 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f., storeName=718cd5d8d8fa4a8a2d92da64e6fb813f/B, priority=12, startTime=1679273859453; duration=0sec 2023-03-20 00:57:39,931 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:39,996 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=417 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/7bf54edb9b13413ea1631ef9952dce45 2023-03-20 00:57:40,289 DEBUG [Thread-2573] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x28072326 to 127.0.0.1:60547 2023-03-20 00:57:40,289 DEBUG [Thread-2573] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:57:40,290 DEBUG [Thread-2579] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x65f69c3f to 127.0.0.1:60547 2023-03-20 00:57:40,290 DEBUG [Thread-2579] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:57:40,291 DEBUG [Thread-2577] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7d3df389 to 127.0.0.1:60547 2023-03-20 00:57:40,291 DEBUG [Thread-2577] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:57:40,291 DEBUG [Thread-2571] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6523d347 to 127.0.0.1:60547 2023-03-20 00:57:40,291 DEBUG [Thread-2571] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:57:40,293 DEBUG [Thread-2575] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2d90797d to 127.0.0.1:60547 2023-03-20 00:57:40,293 DEBUG [Thread-2575] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:57:40,316 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. as already flushing 2023-03-20 00:57:40,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:40,317 DEBUG [Thread-2560] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x682c0b9e to 127.0.0.1:60547 2023-03-20 00:57:40,317 DEBUG [Thread-2560] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:57:40,318 DEBUG [Thread-2566] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d7deb76 to 127.0.0.1:60547 2023-03-20 00:57:40,318 DEBUG [Thread-2566] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:57:40,321 DEBUG [Thread-2562] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50d6899d to 127.0.0.1:60547 2023-03-20 00:57:40,321 DEBUG [Thread-2562] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:57:40,327 DEBUG [Thread-2568] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0f4b7807 to 127.0.0.1:60547 2023-03-20 00:57:40,327 DEBUG [Thread-2568] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:57:40,404 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=417 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/0666c4502d6d4908a3bf1b6b83c7e898 2023-03-20 00:57:40,818 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=417 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/9efd12ec6a0c4c86be358d211caecdac 2023-03-20 00:57:40,824 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/7bf54edb9b13413ea1631ef9952dce45 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/7bf54edb9b13413ea1631ef9952dce45 2023-03-20 00:57:40,828 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/7bf54edb9b13413ea1631ef9952dce45, entries=150, sequenceid=417, filesize=11.9 K 2023-03-20 00:57:40,828 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/0666c4502d6d4908a3bf1b6b83c7e898 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/0666c4502d6d4908a3bf1b6b83c7e898 2023-03-20 00:57:40,835 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/0666c4502d6d4908a3bf1b6b83c7e898, entries=150, sequenceid=417, filesize=11.9 K 2023-03-20 00:57:40,836 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/9efd12ec6a0c4c86be358d211caecdac as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/9efd12ec6a0c4c86be358d211caecdac 2023-03-20 00:57:40,841 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/9efd12ec6a0c4c86be358d211caecdac, entries=150, sequenceid=417, filesize=11.9 K 2023-03-20 00:57:40,843 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=26.84 KB/27480 for 718cd5d8d8fa4a8a2d92da64e6fb813f in 1255ms, sequenceid=417, compaction requested=false 2023-03-20 00:57:40,843 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:40,843 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:40,843 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:40,843 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:40,843 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:40,843 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:40,843 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:40,844 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:40,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:40,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:40,845 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:40,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:40,845 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:40,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:40,845 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:40,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:40,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:40,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:40,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:40,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:40,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25006e52[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:40,850 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:40,850 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:40,851 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:40,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:40,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:40,851 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:40,852 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:40,852 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:40,852 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:40,852 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:40,853 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:40,853 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:40,853 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:40,853 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:40,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:40,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:40,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:40,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:40,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:40,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:40,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:40,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:40,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:40,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:40,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:40,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:40,856 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:40,856 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:40,856 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:40,856 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:40,856 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:40,857 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:40,857 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:40,857 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:40,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:40,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:40,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:40,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:40,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:40,857 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:40,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:40,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:40,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:40,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:40,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:40,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:40,858 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:40,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:40,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:40,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:40,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:40,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:40,860 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:40,860 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:40,860 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:40,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:40,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:40,860 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:40,860 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:40,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:40,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:40,860 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:40,860 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:40,860 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:40,860 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:40,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:40,860 DEBUG [Thread-2570] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:40,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:40,860 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:40,860 DEBUG [Thread-2570] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:40,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:40,961 DEBUG [Thread-2570] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:40,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:42,968 DEBUG [Thread-2564] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6662e946 to 127.0.0.1:60547 2023-03-20 00:57:42,968 DEBUG [Thread-2564] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:57:42,968 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-20 00:57:42,969 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(394): wrote 72 2023-03-20 00:57:42,969 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(394): wrote 70 2023-03-20 00:57:42,969 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(394): wrote 51 2023-03-20 00:57:42,969 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(394): wrote 67 2023-03-20 00:57:42,969 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(394): wrote 71 2023-03-20 00:57:42,969 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-20 00:57:42,969 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-20 00:57:42,969 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(402): scanned 2015 2023-03-20 00:57:42,969 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(403): verified 6045 rows 2023-03-20 00:57:42,969 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(402): scanned 2016 2023-03-20 00:57:42,969 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(403): verified 6044 rows 2023-03-20 00:57:42,969 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(402): scanned 2015 2023-03-20 00:57:42,969 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(403): verified 6043 rows 2023-03-20 00:57:42,969 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(402): scanned 1999 2023-03-20 00:57:42,970 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(403): verified 5997 rows 2023-03-20 00:57:42,970 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(402): scanned 2014 2023-03-20 00:57:42,970 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(403): verified 6040 rows 2023-03-20 00:57:42,970 INFO [Listener at localhost.localdomain/43361] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-03-20 00:57:42,970 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51ff7cd4 to 127.0.0.1:60547 2023-03-20 00:57:42,970 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:57:42,975 INFO [Listener at localhost.localdomain/43361] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-20 00:57:42,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-03-20 00:57:42,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-20 00:57:42,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-20 00:57:42,981 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679273862981"}]},"ts":"1679273862981"} 2023-03-20 00:57:42,983 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-20 00:57:43,075 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-20 00:57:43,076 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=718cd5d8d8fa4a8a2d92da64e6fb813f, UNASSIGN}] 2023-03-20 00:57:43,077 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=718cd5d8d8fa4a8a2d92da64e6fb813f, UNASSIGN 2023-03-20 00:57:43,078 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=718cd5d8d8fa4a8a2d92da64e6fb813f, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:43,079 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273863078"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679273863078"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679273863078"}]},"ts":"1679273863078"} 2023-03-20 00:57:43,080 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199}] 2023-03-20 00:57:43,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-20 00:57:43,233 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:43,233 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 718cd5d8d8fa4a8a2d92da64e6fb813f, disabling compactions & flushes 2023-03-20 00:57:43,233 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:43,233 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:43,233 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. after waiting 0 ms 2023-03-20 00:57:43,234 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:43,234 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 718cd5d8d8fa4a8a2d92da64e6fb813f 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-20 00:57:43,234 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=A 2023-03-20 00:57:43,234 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:43,234 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=B 2023-03-20 00:57:43,235 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:43,235 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 718cd5d8d8fa4a8a2d92da64e6fb813f, store=C 2023-03-20 00:57:43,235 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:43,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-20 00:57:43,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-20 00:57:43,647 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=428 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/028114a4586b49f193823157616d5134 2023-03-20 00:57:44,067 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=428 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/8a2e308ac0084766b4aef1bcf83d420c 2023-03-20 00:57:44,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-20 00:57:44,482 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=428 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/b703697d77c441d490d86bd8d703aa79 2023-03-20 00:57:44,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/A/028114a4586b49f193823157616d5134 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/028114a4586b49f193823157616d5134 2023-03-20 00:57:44,489 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/028114a4586b49f193823157616d5134, entries=150, sequenceid=428, filesize=11.9 K 2023-03-20 00:57:44,489 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/B/8a2e308ac0084766b4aef1bcf83d420c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/8a2e308ac0084766b4aef1bcf83d420c 2023-03-20 00:57:44,492 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/8a2e308ac0084766b4aef1bcf83d420c, entries=150, sequenceid=428, filesize=11.9 K 2023-03-20 00:57:44,493 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/.tmp/C/b703697d77c441d490d86bd8d703aa79 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/b703697d77c441d490d86bd8d703aa79 2023-03-20 00:57:44,496 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/b703697d77c441d490d86bd8d703aa79, entries=150, sequenceid=428, filesize=11.9 K 2023-03-20 00:57:44,497 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 718cd5d8d8fa4a8a2d92da64e6fb813f in 1263ms, sequenceid=428, compaction requested=true 2023-03-20 00:57:44,498 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/69129982a8c34a629d9eba3c2f114346, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/f4ef8b79b1084bd292623ed21d268edc, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/e310e63ce8a2403580b0463c8c3652ed, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/a3f403b0218145ddbcc6a1b50c80d8ba, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/ef707cfc0d5c42f5840f6663fd4ef206, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/fa3a3c378c6040918c085aada9947f96, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/4be38a2b23c74bd2a4c9dc97518e0cd3, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/f6b265cbd23b452fbe148fe2fcc5137c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/ef96b31236bd427cbaa1b61babe61531, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/1137b78030044abf9aa3b39d2a2fcef7, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/7cb25caf8ff64766a1ad244162843727, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/2e545e9bd09741cca18d97fdf7d08a0b, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/a4d62aa425a34b8d8498a4491da9bf6a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/44418d2305964c32b1ec416c15ebaf18, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/319426495254491a8431e02909952f2a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/12ce6edebe224ca993a9ae8bb00fd1af, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/473d1f8c260f4ff2980ae97c28592c42, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/18b8de7b98404aae87dd7eb03ddc29a8, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/ffceb903629d464daf6062282ab23066, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/49d8e83ac383469ba96d842f84ab06ac, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/945741f141c44991bbffbab870ff843f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/4f0a005216814c298de5d7751e93cbf7, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/cfa97321cf044535875ae344df115e0b] to archive 2023-03-20 00:57:44,498 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 00:57:44,499 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/69129982a8c34a629d9eba3c2f114346 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/69129982a8c34a629d9eba3c2f114346 2023-03-20 00:57:44,500 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/f4ef8b79b1084bd292623ed21d268edc to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/f4ef8b79b1084bd292623ed21d268edc 2023-03-20 00:57:44,500 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/e310e63ce8a2403580b0463c8c3652ed to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/e310e63ce8a2403580b0463c8c3652ed 2023-03-20 00:57:44,501 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/a3f403b0218145ddbcc6a1b50c80d8ba to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/a3f403b0218145ddbcc6a1b50c80d8ba 2023-03-20 00:57:44,502 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/ef707cfc0d5c42f5840f6663fd4ef206 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/ef707cfc0d5c42f5840f6663fd4ef206 2023-03-20 00:57:44,502 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/fa3a3c378c6040918c085aada9947f96 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/fa3a3c378c6040918c085aada9947f96 2023-03-20 00:57:44,503 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/4be38a2b23c74bd2a4c9dc97518e0cd3 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/4be38a2b23c74bd2a4c9dc97518e0cd3 2023-03-20 00:57:44,504 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/f6b265cbd23b452fbe148fe2fcc5137c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/f6b265cbd23b452fbe148fe2fcc5137c 2023-03-20 00:57:44,505 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/ef96b31236bd427cbaa1b61babe61531 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/ef96b31236bd427cbaa1b61babe61531 2023-03-20 00:57:44,507 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/1137b78030044abf9aa3b39d2a2fcef7 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/1137b78030044abf9aa3b39d2a2fcef7 2023-03-20 00:57:44,509 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/7cb25caf8ff64766a1ad244162843727 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/7cb25caf8ff64766a1ad244162843727 2023-03-20 00:57:44,510 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/2e545e9bd09741cca18d97fdf7d08a0b to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/2e545e9bd09741cca18d97fdf7d08a0b 2023-03-20 00:57:44,512 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/a4d62aa425a34b8d8498a4491da9bf6a to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/a4d62aa425a34b8d8498a4491da9bf6a 2023-03-20 00:57:44,513 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/44418d2305964c32b1ec416c15ebaf18 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/44418d2305964c32b1ec416c15ebaf18 2023-03-20 00:57:44,513 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/319426495254491a8431e02909952f2a to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/319426495254491a8431e02909952f2a 2023-03-20 00:57:44,514 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/12ce6edebe224ca993a9ae8bb00fd1af to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/12ce6edebe224ca993a9ae8bb00fd1af 2023-03-20 00:57:44,515 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/473d1f8c260f4ff2980ae97c28592c42 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/473d1f8c260f4ff2980ae97c28592c42 2023-03-20 00:57:44,516 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/18b8de7b98404aae87dd7eb03ddc29a8 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/18b8de7b98404aae87dd7eb03ddc29a8 2023-03-20 00:57:44,517 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/ffceb903629d464daf6062282ab23066 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/ffceb903629d464daf6062282ab23066 2023-03-20 00:57:44,519 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/49d8e83ac383469ba96d842f84ab06ac to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/49d8e83ac383469ba96d842f84ab06ac 2023-03-20 00:57:44,520 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/945741f141c44991bbffbab870ff843f to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/945741f141c44991bbffbab870ff843f 2023-03-20 00:57:44,521 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/4f0a005216814c298de5d7751e93cbf7 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/4f0a005216814c298de5d7751e93cbf7 2023-03-20 00:57:44,522 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/cfa97321cf044535875ae344df115e0b to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/cfa97321cf044535875ae344df115e0b 2023-03-20 00:57:44,526 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/b9cdbc543018454d9a6aa4e70d5b224e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/7f9d98ce5d2943c4a80820d80c66fbe9, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/f9ff631ba25d43e29c208a9768745611, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/156c5ade8e0d404eb28d5dba0469d774, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/823830463d4b4386bae7ff42c6be87ba, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/45c9b2fef5e74443951a7e342b8777d3, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/9053f60d3ffc425ca9a91a1443efecbd, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/f178f899704446e883a8b1df59589743, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/1948a5cb8f8e4a51ace11109d49cff54, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/6324a30282fe41eeaa7f18ed7410f10b, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/c47d83db1cbc4daa924c33d92debce6d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/7759bbe5d17647a5974f880c3465954e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/16c63abdcf0142f8ad4f91e6a1789f8e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/f3cd99ee2f344844be447bebc0574c04, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/f5f4622fee0b4075acc91ac9dd0c1daf, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/6aba4217782143399a76a7c7c5e4635a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/726ca955572b453e9b0545c1fbd32007, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/4a95275e27d04cb0abf994f722c96c82, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/59fd2302c6524eeea2c618dfa6bdc797, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/8dcf1b6e5a244310a715cc31d05b4e15, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/0c490aae7ec949299bb281fc08f3041f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/20c29cd560634c2489960bd27da83642, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/7ee586579d834836a4e361bc9a2741ff] to archive 2023-03-20 00:57:44,527 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 00:57:44,528 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/b9cdbc543018454d9a6aa4e70d5b224e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/b9cdbc543018454d9a6aa4e70d5b224e 2023-03-20 00:57:44,529 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/7f9d98ce5d2943c4a80820d80c66fbe9 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/7f9d98ce5d2943c4a80820d80c66fbe9 2023-03-20 00:57:44,530 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/f9ff631ba25d43e29c208a9768745611 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/f9ff631ba25d43e29c208a9768745611 2023-03-20 00:57:44,531 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/156c5ade8e0d404eb28d5dba0469d774 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/156c5ade8e0d404eb28d5dba0469d774 2023-03-20 00:57:44,532 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/823830463d4b4386bae7ff42c6be87ba to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/823830463d4b4386bae7ff42c6be87ba 2023-03-20 00:57:44,532 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/45c9b2fef5e74443951a7e342b8777d3 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/45c9b2fef5e74443951a7e342b8777d3 2023-03-20 00:57:44,533 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/9053f60d3ffc425ca9a91a1443efecbd to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/9053f60d3ffc425ca9a91a1443efecbd 2023-03-20 00:57:44,534 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/f178f899704446e883a8b1df59589743 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/f178f899704446e883a8b1df59589743 2023-03-20 00:57:44,535 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/1948a5cb8f8e4a51ace11109d49cff54 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/1948a5cb8f8e4a51ace11109d49cff54 2023-03-20 00:57:44,535 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/6324a30282fe41eeaa7f18ed7410f10b to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/6324a30282fe41eeaa7f18ed7410f10b 2023-03-20 00:57:44,536 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/c47d83db1cbc4daa924c33d92debce6d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/c47d83db1cbc4daa924c33d92debce6d 2023-03-20 00:57:44,537 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/7759bbe5d17647a5974f880c3465954e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/7759bbe5d17647a5974f880c3465954e 2023-03-20 00:57:44,537 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/16c63abdcf0142f8ad4f91e6a1789f8e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/16c63abdcf0142f8ad4f91e6a1789f8e 2023-03-20 00:57:44,538 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/f3cd99ee2f344844be447bebc0574c04 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/f3cd99ee2f344844be447bebc0574c04 2023-03-20 00:57:44,539 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/f5f4622fee0b4075acc91ac9dd0c1daf to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/f5f4622fee0b4075acc91ac9dd0c1daf 2023-03-20 00:57:44,539 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/6aba4217782143399a76a7c7c5e4635a to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/6aba4217782143399a76a7c7c5e4635a 2023-03-20 00:57:44,540 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/726ca955572b453e9b0545c1fbd32007 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/726ca955572b453e9b0545c1fbd32007 2023-03-20 00:57:44,541 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/4a95275e27d04cb0abf994f722c96c82 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/4a95275e27d04cb0abf994f722c96c82 2023-03-20 00:57:44,541 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/59fd2302c6524eeea2c618dfa6bdc797 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/59fd2302c6524eeea2c618dfa6bdc797 2023-03-20 00:57:44,542 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/8dcf1b6e5a244310a715cc31d05b4e15 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/8dcf1b6e5a244310a715cc31d05b4e15 2023-03-20 00:57:44,543 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/0c490aae7ec949299bb281fc08f3041f to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/0c490aae7ec949299bb281fc08f3041f 2023-03-20 00:57:44,544 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/20c29cd560634c2489960bd27da83642 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/20c29cd560634c2489960bd27da83642 2023-03-20 00:57:44,544 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/7ee586579d834836a4e361bc9a2741ff to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/7ee586579d834836a4e361bc9a2741ff 2023-03-20 00:57:44,547 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/132109c216d34c63ba19e48566e93b65, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/ebe80fa3aad044918d8b4c178d16e95d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/3fa5ca8b189c4482a8131e881c5110fb, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/fd895a9e8fca4764bc618c67a1042ef2, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/7879aeee113e4986b94baf09d505d3c6, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/9bdcddf4aa6b4dabbf23e0811d9c54ff, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/c1065023417c45a7810279a25a1e566b, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/4b5d4373b9c0478f9c90f117a164bf57, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/ea750c050a694f9985950023a0675b86, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/a0a43deccb39400f864456740015f3d3, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/2cba857c4890448fa4ae96c18e42f5d9, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/1f9704b490b64e90b96655c22b7d5cd6, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/d0235b61b1da46bc90529d6192184b66, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/6bd281c0f57045a6ba17985bba36178d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/2197f4a13d7e4508ac39fb1ffc104cfd, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/6f45ad2a8672434fa7221424e91892c2, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/80d95068393348089fd6893e42c13baa, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/b5a7ebe6381f4f1ba48bd6d9f9a348d2, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/2c9fdf0fdf0b43ca8882a62553804d29, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/08f9e5e956c24ca6b26aa48761830dc5, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/52db05dd9c9b43e99e2aa540ad7bb44c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/298129c8a2be439aadf2424535f78ef3, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/674e416095cd45bdb8306a31f812d364] to archive 2023-03-20 00:57:44,547 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 00:57:44,548 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/132109c216d34c63ba19e48566e93b65 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/132109c216d34c63ba19e48566e93b65 2023-03-20 00:57:44,549 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/ebe80fa3aad044918d8b4c178d16e95d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/ebe80fa3aad044918d8b4c178d16e95d 2023-03-20 00:57:44,550 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/3fa5ca8b189c4482a8131e881c5110fb to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/3fa5ca8b189c4482a8131e881c5110fb 2023-03-20 00:57:44,551 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/fd895a9e8fca4764bc618c67a1042ef2 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/fd895a9e8fca4764bc618c67a1042ef2 2023-03-20 00:57:44,551 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/7879aeee113e4986b94baf09d505d3c6 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/7879aeee113e4986b94baf09d505d3c6 2023-03-20 00:57:44,552 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/9bdcddf4aa6b4dabbf23e0811d9c54ff to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/9bdcddf4aa6b4dabbf23e0811d9c54ff 2023-03-20 00:57:44,553 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/c1065023417c45a7810279a25a1e566b to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/c1065023417c45a7810279a25a1e566b 2023-03-20 00:57:44,554 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/4b5d4373b9c0478f9c90f117a164bf57 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/4b5d4373b9c0478f9c90f117a164bf57 2023-03-20 00:57:44,554 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/ea750c050a694f9985950023a0675b86 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/ea750c050a694f9985950023a0675b86 2023-03-20 00:57:44,555 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/a0a43deccb39400f864456740015f3d3 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/a0a43deccb39400f864456740015f3d3 2023-03-20 00:57:44,556 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/2cba857c4890448fa4ae96c18e42f5d9 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/2cba857c4890448fa4ae96c18e42f5d9 2023-03-20 00:57:44,557 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/1f9704b490b64e90b96655c22b7d5cd6 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/1f9704b490b64e90b96655c22b7d5cd6 2023-03-20 00:57:44,558 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/d0235b61b1da46bc90529d6192184b66 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/d0235b61b1da46bc90529d6192184b66 2023-03-20 00:57:44,558 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/6bd281c0f57045a6ba17985bba36178d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/6bd281c0f57045a6ba17985bba36178d 2023-03-20 00:57:44,559 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/2197f4a13d7e4508ac39fb1ffc104cfd to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/2197f4a13d7e4508ac39fb1ffc104cfd 2023-03-20 00:57:44,560 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/6f45ad2a8672434fa7221424e91892c2 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/6f45ad2a8672434fa7221424e91892c2 2023-03-20 00:57:44,560 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/80d95068393348089fd6893e42c13baa to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/80d95068393348089fd6893e42c13baa 2023-03-20 00:57:44,561 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/b5a7ebe6381f4f1ba48bd6d9f9a348d2 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/b5a7ebe6381f4f1ba48bd6d9f9a348d2 2023-03-20 00:57:44,562 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/2c9fdf0fdf0b43ca8882a62553804d29 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/2c9fdf0fdf0b43ca8882a62553804d29 2023-03-20 00:57:44,562 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/08f9e5e956c24ca6b26aa48761830dc5 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/08f9e5e956c24ca6b26aa48761830dc5 2023-03-20 00:57:44,563 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/52db05dd9c9b43e99e2aa540ad7bb44c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/52db05dd9c9b43e99e2aa540ad7bb44c 2023-03-20 00:57:44,564 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/298129c8a2be439aadf2424535f78ef3 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/298129c8a2be439aadf2424535f78ef3 2023-03-20 00:57:44,565 DEBUG [StoreCloser-TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/674e416095cd45bdb8306a31f812d364 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/674e416095cd45bdb8306a31f812d364 2023-03-20 00:57:44,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/recovered.edits/431.seqid, newMaxSeqId=431, maxSeqId=1 2023-03-20 00:57:44,569 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f. 2023-03-20 00:57:44,569 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 718cd5d8d8fa4a8a2d92da64e6fb813f: 2023-03-20 00:57:44,571 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:44,571 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=718cd5d8d8fa4a8a2d92da64e6fb813f, regionState=CLOSED 2023-03-20 00:57:44,571 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273864571"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679273864571"}]},"ts":"1679273864571"} 2023-03-20 00:57:44,573 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-03-20 00:57:44,573 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 718cd5d8d8fa4a8a2d92da64e6fb813f, server=jenkins-hbase20.apache.org,34553,1679273725199 in 1.4920 sec 2023-03-20 00:57:44,574 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-03-20 00:57:44,574 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=718cd5d8d8fa4a8a2d92da64e6fb813f, UNASSIGN in 1.4970 sec 2023-03-20 00:57:44,574 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679273864574"}]},"ts":"1679273864574"} 2023-03-20 00:57:44,575 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-20 00:57:44,576 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-20 00:57:44,577 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6010 sec 2023-03-20 00:57:45,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-20 00:57:45,088 INFO [Listener at localhost.localdomain/43361] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-03-20 00:57:45,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-03-20 00:57:45,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:57:45,094 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:57:45,096 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:57:45,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-20 00:57:45,105 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:45,108 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A, FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B, FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C, FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/recovered.edits] 2023-03-20 00:57:45,112 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/028114a4586b49f193823157616d5134 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/028114a4586b49f193823157616d5134 2023-03-20 00:57:45,113 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/7bf54edb9b13413ea1631ef9952dce45 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/7bf54edb9b13413ea1631ef9952dce45 2023-03-20 00:57:45,115 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/edcf3f9c8eeb48508ae6907414df41a0 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/A/edcf3f9c8eeb48508ae6907414df41a0 2023-03-20 00:57:45,118 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/0666c4502d6d4908a3bf1b6b83c7e898 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/0666c4502d6d4908a3bf1b6b83c7e898 2023-03-20 00:57:45,119 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/8a2e308ac0084766b4aef1bcf83d420c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/8a2e308ac0084766b4aef1bcf83d420c 2023-03-20 00:57:45,120 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/be5f93b8f885494faf2ddedfe953cef0 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/B/be5f93b8f885494faf2ddedfe953cef0 2023-03-20 00:57:45,122 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/73aef888c2014d35a262a4a17a6eaa22 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/73aef888c2014d35a262a4a17a6eaa22 2023-03-20 00:57:45,124 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/9efd12ec6a0c4c86be358d211caecdac to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/9efd12ec6a0c4c86be358d211caecdac 2023-03-20 00:57:45,125 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/b703697d77c441d490d86bd8d703aa79 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/C/b703697d77c441d490d86bd8d703aa79 2023-03-20 00:57:45,128 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/recovered.edits/431.seqid to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f/recovered.edits/431.seqid 2023-03-20 00:57:45,128 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/718cd5d8d8fa4a8a2d92da64e6fb813f 2023-03-20 00:57:45,128 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-20 00:57:45,130 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:57:45,132 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-20 00:57:45,134 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-20 00:57:45,135 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:57:45,135 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-20 00:57:45,136 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679273865135"}]},"ts":"9223372036854775807"} 2023-03-20 00:57:45,138 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-20 00:57:45,138 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 718cd5d8d8fa4a8a2d92da64e6fb813f, NAME => 'TestAcidGuarantees,,1679273838080.718cd5d8d8fa4a8a2d92da64e6fb813f.', STARTKEY => '', ENDKEY => ''}] 2023-03-20 00:57:45,138 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-20 00:57:45,138 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679273865138"}]},"ts":"9223372036854775807"} 2023-03-20 00:57:45,141 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-20 00:57:45,143 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:57:45,144 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 53 msec 2023-03-20 00:57:45,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-20 00:57:45,198 INFO [Listener at localhost.localdomain/43361] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-03-20 00:57:45,213 INFO [Listener at localhost.localdomain/43361] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227 (was 227), OpenFileDescriptor=551 (was 531) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=532 (was 594), ProcessCount=168 (was 168), AvailableMemoryMB=4981 (was 4987) 2023-03-20 00:57:45,226 INFO [Listener at localhost.localdomain/43361] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=227, OpenFileDescriptor=551, MaxFileDescriptor=60000, SystemLoadAverage=532, ProcessCount=168, AvailableMemoryMB=4980 2023-03-20 00:57:45,228 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-20 00:57:45,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-20 00:57:45,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-20 00:57:45,230 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-20 00:57:45,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-03-20 00:57:45,231 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-20 00:57:45,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-20 00:57:45,232 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:45,233 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632 empty. 2023-03-20 00:57:45,233 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:45,233 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-20 00:57:45,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-20 00:57:45,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-20 00:57:45,642 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-20 00:57:45,644 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 3796a7d329c46cabe9e95c4c1fb73632, NAME => 'TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp 2023-03-20 00:57:45,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-20 00:57:46,055 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 00:57:46,055 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 3796a7d329c46cabe9e95c4c1fb73632, disabling compactions & flushes 2023-03-20 00:57:46,055 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:46,055 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:46,055 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. after waiting 0 ms 2023-03-20 00:57:46,055 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:46,055 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:46,055 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:57:46,057 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-20 00:57:46,058 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273866058"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679273866058"}]},"ts":"1679273866058"} 2023-03-20 00:57:46,060 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-20 00:57:46,061 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-20 00:57:46,061 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679273866061"}]},"ts":"1679273866061"} 2023-03-20 00:57:46,062 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-20 00:57:46,064 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3796a7d329c46cabe9e95c4c1fb73632, ASSIGN}] 2023-03-20 00:57:46,066 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3796a7d329c46cabe9e95c4c1fb73632, ASSIGN 2023-03-20 00:57:46,067 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=3796a7d329c46cabe9e95c4c1fb73632, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34553,1679273725199; forceNewPlan=false, retain=false 2023-03-20 00:57:46,218 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=3796a7d329c46cabe9e95c4c1fb73632, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:46,219 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273866218"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679273866218"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679273866218"}]},"ts":"1679273866218"} 2023-03-20 00:57:46,220 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199}] 2023-03-20 00:57:46,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-20 00:57:46,377 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:46,377 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 3796a7d329c46cabe9e95c4c1fb73632, NAME => 'TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.', STARTKEY => '', ENDKEY => ''} 2023-03-20 00:57:46,377 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:46,377 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 00:57:46,377 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:46,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:46,378 INFO [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:46,379 DEBUG [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A 2023-03-20 00:57:46,380 DEBUG [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A 2023-03-20 00:57:46,380 INFO [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 00:57:46,380 INFO [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3796a7d329c46cabe9e95c4c1fb73632 columnFamilyName A 2023-03-20 00:57:46,380 INFO [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] regionserver.HStore(310): Store=3796a7d329c46cabe9e95c4c1fb73632/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:57:46,381 INFO [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:46,382 DEBUG [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B 2023-03-20 00:57:46,382 DEBUG [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B 2023-03-20 00:57:46,382 INFO [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 00:57:46,382 INFO [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3796a7d329c46cabe9e95c4c1fb73632 columnFamilyName B 2023-03-20 00:57:46,383 INFO [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] regionserver.HStore(310): Store=3796a7d329c46cabe9e95c4c1fb73632/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:57:46,383 INFO [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:46,384 DEBUG [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C 2023-03-20 00:57:46,384 DEBUG [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C 2023-03-20 00:57:46,384 INFO [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 00:57:46,384 INFO [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3796a7d329c46cabe9e95c4c1fb73632 columnFamilyName C 2023-03-20 00:57:46,385 INFO [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] regionserver.HStore(310): Store=3796a7d329c46cabe9e95c4c1fb73632/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:57:46,385 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:46,385 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:46,386 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:46,387 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-20 00:57:46,388 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:46,390 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-20 00:57:46,391 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 3796a7d329c46cabe9e95c4c1fb73632; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69220317, jitterRate=0.031463101506233215}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-20 00:57:46,391 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:57:46,392 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632., pid=56, masterSystemTime=1679273866373 2023-03-20 00:57:46,393 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:46,393 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:46,393 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=3796a7d329c46cabe9e95c4c1fb73632, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:46,393 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273866393"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679273866393"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679273866393"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679273866393"}]},"ts":"1679273866393"} 2023-03-20 00:57:46,395 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-03-20 00:57:46,395 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 in 174 msec 2023-03-20 00:57:46,397 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-03-20 00:57:46,397 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3796a7d329c46cabe9e95c4c1fb73632, ASSIGN in 332 msec 2023-03-20 00:57:46,398 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-20 00:57:46,398 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679273866398"}]},"ts":"1679273866398"} 2023-03-20 00:57:46,399 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-20 00:57:46,402 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-20 00:57:46,403 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1740 sec 2023-03-20 00:57:47,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-20 00:57:47,340 INFO [Listener at localhost.localdomain/43361] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-03-20 00:57:47,340 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x5be55982 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:57:47,346 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@630ea023, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:57:47,348 DEBUG [Listener at localhost.localdomain/43361] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:57:47,349 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49424, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:57:47,350 DEBUG [Listener at localhost.localdomain/43361] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-20 00:57:47,353 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:46946, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-20 00:57:47,354 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-20 00:57:47,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-20 00:57:47,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-20 00:57:47,368 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-20 00:57:47,369 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-20 00:57:47,369 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-20 00:57:47,371 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-20 00:57:47,374 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3796a7d329c46cabe9e95c4c1fb73632, REOPEN/MOVE}] 2023-03-20 00:57:47,375 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3796a7d329c46cabe9e95c4c1fb73632, REOPEN/MOVE 2023-03-20 00:57:47,376 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=3796a7d329c46cabe9e95c4c1fb73632, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:47,376 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273867376"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679273867376"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679273867376"}]},"ts":"1679273867376"} 2023-03-20 00:57:47,377 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199}] 2023-03-20 00:57:47,529 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:47,531 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 3796a7d329c46cabe9e95c4c1fb73632, disabling compactions & flushes 2023-03-20 00:57:47,531 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:47,531 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:47,531 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. after waiting 0 ms 2023-03-20 00:57:47,531 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:47,539 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-20 00:57:47,540 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:47,540 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:57:47,540 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 3796a7d329c46cabe9e95c4c1fb73632 to self. 2023-03-20 00:57:47,542 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:47,543 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=3796a7d329c46cabe9e95c4c1fb73632, regionState=CLOSED 2023-03-20 00:57:47,543 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273867543"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679273867543"}]},"ts":"1679273867543"} 2023-03-20 00:57:47,546 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-03-20 00:57:47,546 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 in 167 msec 2023-03-20 00:57:47,546 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=3796a7d329c46cabe9e95c4c1fb73632, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,34553,1679273725199; forceNewPlan=false, retain=true 2023-03-20 00:57:47,697 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=3796a7d329c46cabe9e95c4c1fb73632, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:47,697 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273867697"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679273867697"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679273867697"}]},"ts":"1679273867697"} 2023-03-20 00:57:47,698 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199}] 2023-03-20 00:57:47,855 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:47,855 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 3796a7d329c46cabe9e95c4c1fb73632, NAME => 'TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.', STARTKEY => '', ENDKEY => ''} 2023-03-20 00:57:47,856 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:47,856 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 00:57:47,856 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:47,856 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:47,858 INFO [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:47,860 DEBUG [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A 2023-03-20 00:57:47,860 DEBUG [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A 2023-03-20 00:57:47,860 INFO [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 00:57:47,860 INFO [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3796a7d329c46cabe9e95c4c1fb73632 columnFamilyName A 2023-03-20 00:57:47,862 INFO [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] regionserver.HStore(310): Store=3796a7d329c46cabe9e95c4c1fb73632/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:57:47,863 INFO [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:47,864 DEBUG [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B 2023-03-20 00:57:47,864 DEBUG [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B 2023-03-20 00:57:47,864 INFO [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 00:57:47,864 INFO [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3796a7d329c46cabe9e95c4c1fb73632 columnFamilyName B 2023-03-20 00:57:47,865 INFO [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] regionserver.HStore(310): Store=3796a7d329c46cabe9e95c4c1fb73632/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:57:47,865 INFO [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:47,866 DEBUG [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C 2023-03-20 00:57:47,866 DEBUG [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C 2023-03-20 00:57:47,867 INFO [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 00:57:47,867 INFO [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3796a7d329c46cabe9e95c4c1fb73632 columnFamilyName C 2023-03-20 00:57:47,867 INFO [StoreOpener-3796a7d329c46cabe9e95c4c1fb73632-1] regionserver.HStore(310): Store=3796a7d329c46cabe9e95c4c1fb73632/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 00:57:47,868 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:47,869 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:47,870 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:47,873 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-20 00:57:47,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:47,875 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 3796a7d329c46cabe9e95c4c1fb73632; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62230456, jitterRate=-0.07269394397735596}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-20 00:57:47,875 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:57:47,875 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632., pid=61, masterSystemTime=1679273867850 2023-03-20 00:57:47,876 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:47,876 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:47,877 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=3796a7d329c46cabe9e95c4c1fb73632, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:47,877 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273867877"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679273867877"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679273867877"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679273867877"}]},"ts":"1679273867877"} 2023-03-20 00:57:47,879 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-03-20 00:57:47,879 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 in 180 msec 2023-03-20 00:57:47,880 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-03-20 00:57:47,880 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3796a7d329c46cabe9e95c4c1fb73632, REOPEN/MOVE in 505 msec 2023-03-20 00:57:47,881 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-03-20 00:57:47,882 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 509 msec 2023-03-20 00:57:47,883 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 527 msec 2023-03-20 00:57:47,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-03-20 00:57:47,885 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x2f11e84a to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:57:47,891 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ec98d63, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:57:47,892 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x3a1010a6 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:57:47,896 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ae3ec16, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:57:47,897 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x4143b9ae to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:57:47,917 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7bec0400, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:57:47,918 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x686af57e to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:57:47,932 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34b8bf14, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:57:47,933 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x078cd331 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:57:47,942 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@387217bb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:57:47,942 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x7c00a18c to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:57:47,956 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5467be78, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:57:47,957 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x3da68fe0 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:57:47,965 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44535519, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:57:47,965 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x7fd90ac8 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:57:47,975 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7689435c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:57:47,975 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x70e63396 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:57:47,983 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@475ee808, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:57:47,984 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(139): Connect 0x32cc4808 to 127.0.0.1:60547 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 00:57:47,992 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7377b744, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 00:57:48,007 DEBUG [hconnection-0xf96eeda-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:57:48,008 DEBUG [hconnection-0x466459d7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:57:48,008 DEBUG [hconnection-0x3270d993-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:57:48,008 DEBUG [hconnection-0x33cf02c5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:57:48,008 DEBUG [hconnection-0x65904b8b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:57:48,007 DEBUG [hconnection-0x39cbd810-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:57:48,009 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49440, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:57:48,008 DEBUG [hconnection-0x56194d37-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:57:48,011 DEBUG [hconnection-0x300ced91-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:57:48,008 DEBUG [hconnection-0x407f4b2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:57:48,011 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49438, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:57:48,011 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49460, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:57:48,010 DEBUG [hconnection-0x57e75a5d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 00:57:48,011 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49492, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:57:48,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:48,014 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49480, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:57:48,015 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49526, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:57:48,017 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49514, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:57:48,018 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49468, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:57:48,023 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49504, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:57:48,024 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49538, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 00:57:48,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:48,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:48,032 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:48,032 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:48,032 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:48,032 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:48,033 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,033 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,034 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:48,034 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:48,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:48,035 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,035 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:48,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:48,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,035 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:48,036 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:48,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:48,037 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:48,037 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3796a7d329c46cabe9e95c4c1fb73632 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-20 00:57:48,037 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:48,037 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:48,037 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:48,037 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=A 2023-03-20 00:57:48,037 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:48,038 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=B 2023-03-20 00:57:48,038 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:48,038 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=C 2023-03-20 00:57:48,038 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:48,038 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:48,038 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:48,038 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:48,038 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:48,038 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:48,038 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:48,038 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:48,038 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:48,038 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,038 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:48,038 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:48,039 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:48,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:48,039 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,040 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,040 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:48,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:48,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7765d2a1[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:48,042 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:48,042 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,043 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,043 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:48,044 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,044 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:48,044 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:48,044 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:48,047 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,047 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:48,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:48,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:48,047 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:48,047 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:48,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:48,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:48,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:48,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:48,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:48,049 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:48,049 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:48,049 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:48,049 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:48,049 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:48,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679273868046, max timestamp=1679273868050Num uniques 859; 2023-03-20 00:57:48,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 00:57:48,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679273868046, max timestamp=1679273868050Num uniques 859; 2023-03-20 00:57:48,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 00:57:48,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679273868046, max timestamp=1679273868050Num uniques 859; 2023-03-20 00:57:48,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 00:57:48,058 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,058 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:48,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:48,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,058 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:48,058 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:48,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:48,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:48,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:48,059 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:48,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,060 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:48,060 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,060 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:48,060 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:48,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:48,060 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:48,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:48,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:48,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:48,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:48,060 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273928058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,061 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:48,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:48,061 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,061 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:48,061 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:48,061 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,061 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:48,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273928058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:48,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49504 deadline: 1679273928059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,062 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:48,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273928060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:48,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273928061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,161 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:48,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:48,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:48,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273928162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:48,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:48,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273928162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,164 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:48,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273928163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,164 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:48,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49504 deadline: 1679273928163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:48,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:48,165 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:48,165 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:48,165 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:48,165 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:48,166 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,166 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,166 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:48,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273928164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,166 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,166 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:48,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:48,167 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,167 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:48,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:48,169 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:48,169 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:48,169 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:48,169 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:48,169 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:48,170 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:48,170 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:48,170 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:48,170 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:48,170 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:48,170 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:48,170 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:48,170 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:48,170 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,170 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:48,170 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:48,173 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,173 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:48,173 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,173 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:48,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:48,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:48,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:48,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:48,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32dfa938[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:48,176 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:48,177 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,177 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,177 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:48,177 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:48,177 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:48,177 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,177 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:48,178 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:48,178 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:48,178 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:48,178 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:48,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:48,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:48,181 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:48,181 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:48,181 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:48,181 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:48,181 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:48,182 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,182 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,182 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:48,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:48,183 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,184 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:48,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,186 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,186 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,186 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:48,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:48,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:48,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:48,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:48,186 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:48,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:48,186 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:48,186 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,186 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,186 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:48,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:48,186 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:48,186 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:48,186 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,187 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,287 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:48,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:48,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:48,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:48,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:48,309 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:48,310 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:48,310 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:48,310 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:48,310 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,310 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,311 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,311 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:48,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:48,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:48,311 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,311 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:48,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:48,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,312 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:48,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:48,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:48,318 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:48,318 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:48,318 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:48,318 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:48,318 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:48,319 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:48,319 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:48,319 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:48,319 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:48,319 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:48,319 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:48,319 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,319 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:48,319 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:48,324 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,324 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:48,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:48,325 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,325 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:48,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:48,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:48,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:48,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:48,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c6c8de2[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:48,326 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:48,326 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,327 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,327 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:48,327 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:48,327 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:48,327 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,327 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:48,327 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:48,327 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,327 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:48,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:48,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:48,328 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:48,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:48,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:48,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:48,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:48,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:48,330 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:48,330 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:48,330 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:48,330 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:48,330 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:48,331 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,331 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:48,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:48,331 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:48,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:48,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:48,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:48,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,332 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:48,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,333 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:48,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,336 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:48,336 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,336 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:48,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:48,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:48,336 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:48,336 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:48,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:48,336 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:48,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:48,336 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:48,336 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:48,336 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,336 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,336 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,336 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:48,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273928364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:48,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273928366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:48,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273928366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:48,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49504 deadline: 1679273928366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:48,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273928368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,436 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:48,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:48,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:48,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:48,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:48,440 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:48,440 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:48,440 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:48,440 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:48,440 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,440 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,441 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:48,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:48,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:48,441 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,441 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:48,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:48,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,442 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:48,442 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:48,442 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:48,442 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:48,442 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:48,442 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:48,442 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:48,442 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:48,442 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:48,442 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:48,442 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:48,442 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:48,442 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:48,442 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:48,442 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,442 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:48,442 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:48,443 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,443 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:48,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:48,443 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,443 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:48,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:48,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:48,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:48,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:48,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a3814c3[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:48,445 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:48,445 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,445 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,445 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:48,446 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,447 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:48,447 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:48,447 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:48,449 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032094447f45f0e74c679a36b94223cb2cde to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032094447f45f0e74c679a36b94223cb2cde 2023-03-20 00:57:48,451 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:48,452 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:48,452 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,452 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:48,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:48,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:48,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:48,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:48,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:48,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:48,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:48,454 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:48,454 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:48,454 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:48,454 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:48,454 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:48,455 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,455 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:48,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:48,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,455 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:48,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:48,456 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:48,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,457 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:48,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,460 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,460 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:48,460 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:48,460 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,460 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:48,460 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:48,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:48,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:48,460 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:48,460 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:48,460 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:48,460 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,461 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,461 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,561 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:48,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:48,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:48,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:48,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:48,582 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:48,582 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:48,582 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:48,582 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:48,583 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,583 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,583 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:48,583 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:48,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:48,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,584 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,584 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:48,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:48,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,585 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:48,585 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:48,585 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:48,585 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:48,585 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:48,594 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:48,594 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:48,594 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:48,594 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:48,594 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:48,594 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:48,594 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:48,594 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:48,594 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,594 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:48,594 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:48,606 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,606 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:48,606 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,606 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:48,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:48,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53912fff[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:48,607 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:48,607 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,608 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,608 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:48,608 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:48,608 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,608 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:48,608 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:48,608 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:48,609 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:48,609 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:48,609 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:48,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:48,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:48,610 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:48,610 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:48,610 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:48,610 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:48,610 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:48,611 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,611 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:48,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,611 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:48,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:48,612 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,613 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:48,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,614 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:48,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:48,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:48,614 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:48,614 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,614 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,614 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:48,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:48,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:48,614 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:48,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:48,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:48,614 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,614 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:48,614 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,614 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:48,614 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,615 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,667 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:48,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273928667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:48,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273928668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:48,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273928669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,672 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:48,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49504 deadline: 1679273928670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:48,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273928671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,715 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:48,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:48,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:48,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:48,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:48,718 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:48,718 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:48,718 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:48,718 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:48,718 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,718 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,719 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,719 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:48,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:48,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:48,719 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,719 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:48,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:48,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:48,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:48,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:48,722 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:48,722 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:48,722 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:48,724 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:48,724 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:48,725 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:48,725 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:48,725 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:48,725 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:48,725 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:48,725 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:48,725 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,725 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:48,725 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:48,733 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,733 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:48,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:48,733 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,733 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:48,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:48,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:48,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:48,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:48,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c48cadc[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:48,734 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:48,734 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,739 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,739 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:48,739 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:48,739 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,739 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:48,739 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:48,740 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:48,740 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:48,740 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,740 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:48,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:48,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:48,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:48,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:48,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:48,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:48,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:48,741 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:48,742 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:48,742 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:48,742 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:48,742 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:48,742 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,742 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:48,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:48,742 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:48,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:48,743 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:48,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,744 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:48,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,745 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,745 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,745 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,745 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:48,745 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:48,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:48,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:48,745 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,745 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,745 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,745 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:48,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:48,745 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:48,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:48,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:48,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:48,745 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:48,746 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:48,846 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:48,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:48,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:48,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:48,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:48,848 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:48,848 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:48,848 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:48,848 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:48,849 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,849 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,849 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,849 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:48,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:48,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:48,850 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,850 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,850 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:48,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:48,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:48,851 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:48,851 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:48,851 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:48,851 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:48,851 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:48,851 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:48,851 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:48,851 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:48,851 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:48,851 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:48,851 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:48,851 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,851 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:48,851 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:48,852 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,852 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:48,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:48,852 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,852 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:48,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:48,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:48,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:48,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:48,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5483472a[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:48,854 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:48,854 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,854 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,855 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:48,855 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:48,855 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:48,855 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,855 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:48,855 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/606efe733faf45cf99fefcc8449533fa 2023-03-20 00:57:48,855 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,855 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:48,855 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:48,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:48,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:48,856 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:48,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:48,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:48,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:48,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:48,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:48,858 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:48,858 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:48,858 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:48,858 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:48,859 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:48,860 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,861 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:48,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,861 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:48,861 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:48,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:48,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:48,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,866 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:48,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,876 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:48,876 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:48,876 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:48,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:48,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:48,876 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:48,876 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,876 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,876 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:48,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:48,876 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:48,876 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,876 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,876 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,876 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:48,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:48,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:48,976 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:48,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:48,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:48,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:48,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:48,985 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:48,985 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:48,985 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:48,985 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:48,985 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,985 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,986 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:48,986 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:48,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:48,986 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,986 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:48,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:48,987 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:48,987 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:48,987 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:48,987 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:48,987 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:48,987 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:48,987 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:48,987 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:48,987 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:48,987 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:48,988 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:48,988 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:48,988 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:48,988 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:48,988 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,988 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:48,988 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:48,989 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,993 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:48,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:48,993 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:48,993 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:48,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:48,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:48,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:48,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:48,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:48,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:48,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e57df75[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:48,994 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:48,994 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,008 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,008 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,008 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:49,008 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:49,008 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:49,008 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:49,014 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,014 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:49,014 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:49,014 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:49,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:49,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:49,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:49,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:49,016 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:49,016 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:49,016 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:49,016 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:49,016 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:49,017 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,017 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,017 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:49,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:49,017 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,019 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,020 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,020 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:49,020 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:49,020 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:49,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:49,020 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:49,020 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:49,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:49,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:49,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:49,020 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,021 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,021 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,021 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:49,021 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:49,021 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:49,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:49,121 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:49,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:49,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:49,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:49,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:49,124 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:49,124 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:49,124 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:49,124 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:49,125 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,125 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:49,125 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,125 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:49,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:49,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:49,126 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,126 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:49,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:49,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:49,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:49,128 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:49,129 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:49,129 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:49,129 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:49,129 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:49,129 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:49,129 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:49,129 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:49,129 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:49,129 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:49,129 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:49,129 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:49,129 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:49,129 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:49,129 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,130 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:49,130 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:49,139 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,139 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:49,140 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,140 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:49,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:49,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44e3e597[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:49,143 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:49,143 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,150 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,151 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,151 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:49,151 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:49,151 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:49,151 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:49,158 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:49,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:49,158 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:49,158 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:49,158 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:49,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:49,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:49,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:49,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:49,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:49,163 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:49,163 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:49,163 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:49,164 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:49,164 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:49,166 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,166 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,166 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:49,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:49,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:49,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:49,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:49,167 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:49,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:49,170 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:49,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:49,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273929170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:49,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273929172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:49,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273929172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,174 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:49,175 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:49,175 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:49,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:49,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:49,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:49,175 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:49,175 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:49,175 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:49,175 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:49,175 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:49,176 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,176 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,176 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:49,176 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:49,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:49,177 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:49,177 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:49,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273929175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49504 deadline: 1679273929175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,266 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/3900c55c86984042ae1c9b7ace407527 2023-03-20 00:57:49,275 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:49,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:49,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:49,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:49,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:49,280 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:49,280 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:49,280 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:49,280 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:49,280 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,280 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:49,281 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,281 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:49,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:49,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:49,281 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,281 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:49,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:49,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:49,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:49,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:49,282 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:49,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:49,282 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:49,282 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:49,282 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:49,282 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:49,282 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:49,282 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:49,282 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:49,282 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:49,282 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:49,282 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:49,282 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:49,282 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,282 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:49,282 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:49,283 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,283 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:49,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:49,283 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,283 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:49,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:49,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:49,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e27d9fd[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:49,285 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:49,285 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,286 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,286 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:49,286 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:49,286 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:49,286 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,286 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:49,287 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,287 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:49,287 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:49,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:49,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:49,287 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:49,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:49,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:49,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:49,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:49,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:49,289 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:49,289 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:49,289 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:49,289 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:49,289 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:49,290 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:49,290 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,290 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:49,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:49,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:49,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,291 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:49,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,292 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:49,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,293 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,293 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:49,293 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:49,293 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:49,293 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:49,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:49,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:49,294 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:49,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:49,293 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:49,294 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,294 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,294 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,294 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:49,294 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:49,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:49,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:49,394 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:49,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:49,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:49,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:49,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:49,397 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:49,397 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:49,397 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:49,397 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:49,397 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,397 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:49,399 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:49,399 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:49,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:49,399 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,399 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:49,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:49,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:49,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:49,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:49,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:49,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:49,400 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:49,400 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:49,400 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:49,400 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:49,400 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:49,400 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:49,400 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:49,400 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:49,400 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:49,400 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:49,400 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:49,400 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,400 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:49,400 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:49,401 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,401 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:49,401 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,401 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:49,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:49,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50d36704[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:49,402 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:49,402 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,403 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,403 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:49,403 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,403 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:49,403 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:49,403 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:49,404 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:49,404 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:49,404 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,404 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:49,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:49,406 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:49,406 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:49,406 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:49,406 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:49,406 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:49,407 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,407 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:49,407 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:49,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:49,408 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,409 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:49,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,410 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,410 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:49,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:49,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:49,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:49,410 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:49,410 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:49,411 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:49,411 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:49,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:49,411 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:49,411 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:49,411 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,411 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,411 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,411 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:49,511 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:49,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:49,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:49,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:49,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:49,513 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:49,513 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:49,514 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:49,514 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:49,514 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,514 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:49,514 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,514 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:49,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:49,515 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,515 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:49,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:49,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:49,516 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:49,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:49,516 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:49,516 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:49,516 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:49,516 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:49,516 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:49,516 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:49,516 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:49,516 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:49,516 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:49,516 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:49,516 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:49,516 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,516 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:49,516 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:49,517 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,517 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:49,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:49,517 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,517 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:49,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:49,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:49,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:49,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:49,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70dfa61c[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:49,518 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:49,518 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,519 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,519 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,519 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:49,519 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:49,519 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:49,519 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:49,520 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,520 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:49,520 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:49,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:49,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:49,520 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:49,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:49,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:49,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:49,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:49,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:49,521 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:49,521 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:49,521 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:49,521 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:49,521 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:49,522 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,522 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,522 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:49,522 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:49,523 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:49,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,524 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,524 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:49,524 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:49,524 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:49,524 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:49,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:49,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:49,524 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:49,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:49,525 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:49,525 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:49,524 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:49,525 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,525 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,525 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:49,625 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:49,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:49,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:49,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:49,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:49,628 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:49,628 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:49,628 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:49,628 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:49,628 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,628 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:49,629 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:49,629 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:49,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:49,629 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,629 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:49,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:49,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:49,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:49,630 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:49,630 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:49,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:49,630 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:49,630 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:49,630 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:49,630 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:49,630 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:49,630 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:49,630 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:49,630 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:49,630 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:49,631 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:49,631 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:49,631 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,631 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:49,631 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:49,632 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,632 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:49,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:49,632 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,632 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:49,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:49,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:49,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:49,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b4b3acd[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:49,633 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:49,633 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,634 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,634 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:49,634 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:49,634 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,634 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:49,634 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:49,635 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,635 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:49,635 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:49,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:49,635 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:49,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:49,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:49,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:49,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:49,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:49,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:49,637 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:49,637 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:49,637 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:49,637 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:49,637 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:49,638 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:49,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:49,638 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,638 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:49,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:49,639 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,639 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:49,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,641 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,641 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:49,641 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:49,641 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:49,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:49,641 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:49,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:49,641 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:49,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:49,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:49,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:49,641 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,642 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,642 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:49,642 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,642 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:49,642 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:49,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:49,677 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/04b5de4c1b2e40749030f64ed7e0ba39 2023-03-20 00:57:49,680 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/606efe733faf45cf99fefcc8449533fa as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/606efe733faf45cf99fefcc8449533fa 2023-03-20 00:57:49,683 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/606efe733faf45cf99fefcc8449533fa, entries=150, sequenceid=16, filesize=25.1 K 2023-03-20 00:57:49,684 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/3900c55c86984042ae1c9b7ace407527 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/3900c55c86984042ae1c9b7ace407527 2023-03-20 00:57:49,687 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/3900c55c86984042ae1c9b7ace407527, entries=150, sequenceid=16, filesize=11.6 K 2023-03-20 00:57:49,687 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/04b5de4c1b2e40749030f64ed7e0ba39 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/04b5de4c1b2e40749030f64ed7e0ba39 2023-03-20 00:57:49,690 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/04b5de4c1b2e40749030f64ed7e0ba39, entries=150, sequenceid=16, filesize=11.6 K 2023-03-20 00:57:49,690 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 3796a7d329c46cabe9e95c4c1fb73632 in 1653ms, sequenceid=16, compaction requested=false 2023-03-20 00:57:49,690 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:57:49,742 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:49,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:49,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:49,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:49,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:49,747 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:49,747 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:49,747 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:49,747 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:49,748 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,748 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:49,749 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:49,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:49,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:49,749 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:49,749 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:49,749 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:49,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:49,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:49,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:49,751 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:49,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:49,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:49,751 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:49,751 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:49,751 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:49,753 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:49,753 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:49,753 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:49,753 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:49,753 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3796a7d329c46cabe9e95c4c1fb73632 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-03-20 00:57:49,754 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=A 2023-03-20 00:57:49,754 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:57:49,754 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=B 2023-03-20 00:57:49,754 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:57:49,754 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=C 2023-03-20 00:57:49,754 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:57:49,777 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303208da6b2bba13e428e9499fd8e043377c2 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303208da6b2bba13e428e9499fd8e043377c2 2023-03-20 00:57:49,792 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/930d42d43f8045c7b8c7594b5bb43859 2023-03-20 00:57:49,801 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/46c0d7ada6bb4f9d81455536b8fe5a75 2023-03-20 00:57:50,090 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-20 00:57:50,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:50,180 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:50,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:50,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:50,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273930185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273930185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:50,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:50,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273930186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49504 deadline: 1679273930187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:50,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273930189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,211 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/b787a30d2a344679a3f6a8b89d5075cb 2023-03-20 00:57:50,214 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/930d42d43f8045c7b8c7594b5bb43859 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/930d42d43f8045c7b8c7594b5bb43859 2023-03-20 00:57:50,217 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/930d42d43f8045c7b8c7594b5bb43859, entries=150, sequenceid=50, filesize=25.1 K 2023-03-20 00:57:50,218 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/46c0d7ada6bb4f9d81455536b8fe5a75 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/46c0d7ada6bb4f9d81455536b8fe5a75 2023-03-20 00:57:50,221 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/46c0d7ada6bb4f9d81455536b8fe5a75, entries=150, sequenceid=50, filesize=11.6 K 2023-03-20 00:57:50,222 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/b787a30d2a344679a3f6a8b89d5075cb as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/b787a30d2a344679a3f6a8b89d5075cb 2023-03-20 00:57:50,230 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/b787a30d2a344679a3f6a8b89d5075cb, entries=150, sequenceid=50, filesize=11.6 K 2023-03-20 00:57:50,231 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=60.38 KB/61830 for 3796a7d329c46cabe9e95c4c1fb73632 in 478ms, sequenceid=50, compaction requested=false 2023-03-20 00:57:50,231 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:57:50,231 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:50,231 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:50,231 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:50,231 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,231 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:50,231 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:50,233 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:50,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:50,233 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:50,233 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,233 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:50,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:50,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:50,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:50,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b03e668[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:50,235 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:50,235 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,236 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,236 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:50,236 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:50,236 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:50,236 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,236 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:50,237 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:50,237 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,237 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:50,237 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:50,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:50,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:50,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:50,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:50,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:50,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:50,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:50,239 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:50,239 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:50,239 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:50,240 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:50,240 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:50,240 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,240 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,241 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:50,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:50,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:50,241 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:50,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:50,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:50,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,242 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:50,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:50,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,244 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:50,244 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:50,244 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:50,244 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:50,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:50,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:50,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:50,244 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,244 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,244 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:50,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:50,244 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:50,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:50,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:50,244 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,244 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,244 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,244 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:50,291 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3796a7d329c46cabe9e95c4c1fb73632 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-20 00:57:50,292 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=A 2023-03-20 00:57:50,292 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:50,292 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=B 2023-03-20 00:57:50,292 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:50,292 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=C 2023-03-20 00:57:50,292 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:50,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:50,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273930302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:50,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273930303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,305 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:50,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273930303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,305 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:50,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273930304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,344 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:50,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:50,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:50,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:50,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:50,347 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:50,347 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:50,347 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:50,347 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:50,347 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,347 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,348 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:50,348 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:50,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:50,348 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,348 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:50,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,349 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:50,349 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:50,349 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:50,349 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:50,349 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:50,349 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:50,349 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:50,349 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:50,349 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:50,349 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:50,349 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:50,349 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:50,349 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:50,349 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,349 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:50,349 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:50,350 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,350 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:50,350 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,350 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:50,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:50,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:50,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:50,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40a35b4b[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:50,351 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:50,352 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,352 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,352 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:50,352 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:50,352 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:50,352 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,352 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:50,353 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,353 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:50,353 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:50,353 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:50,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:50,355 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:50,355 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:50,355 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:50,355 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:50,355 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:50,356 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,356 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:50,356 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:50,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:50,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,357 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:50,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,358 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,359 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,359 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,359 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,359 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:50,359 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:50,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:50,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:50,359 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,359 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:50,359 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:50,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:50,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:50,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:50,359 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,359 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,359 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:50,359 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:50,406 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:50,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273930405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,406 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:50,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273930405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:50,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273930405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:50,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273930406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,460 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:50,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:50,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:50,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:50,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:50,462 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:50,462 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:50,462 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:50,462 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:50,462 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,462 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,463 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,463 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:50,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:50,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:50,463 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,463 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:50,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:50,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,464 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:50,464 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:50,464 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:50,464 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:50,464 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:50,464 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:50,464 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:50,464 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:50,464 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:50,465 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:50,465 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:50,465 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:50,465 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:50,465 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:50,465 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,465 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:50,465 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:50,466 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,466 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:50,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:50,466 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,466 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:50,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:50,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:50,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:50,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:50,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dc1ea05[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:50,467 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:50,467 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,468 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,468 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:50,468 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:50,468 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,468 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:50,468 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:50,469 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,469 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:50,469 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:50,469 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:50,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:50,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:50,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:50,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:50,471 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:50,471 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:50,471 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:50,471 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:50,471 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:50,472 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,472 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,472 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:50,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:50,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:50,472 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,473 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:50,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,475 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,475 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:50,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:50,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:50,475 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:50,475 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,475 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,475 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,475 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:50,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:50,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:50,475 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:50,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:50,475 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:50,475 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,475 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,475 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:50,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:50,575 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:50,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:50,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:50,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:50,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:50,582 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:50,582 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:50,582 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:50,582 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:50,582 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,582 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,583 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:50,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:50,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:50,583 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,584 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,584 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:50,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:50,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,584 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:50,589 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:50,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:50,589 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:50,589 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:50,589 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:50,590 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:50,590 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:50,590 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:50,590 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:50,590 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:50,590 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:50,590 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:50,590 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:50,590 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,590 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:50,590 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:50,597 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,597 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:50,598 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,598 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:50,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:50,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:50,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:50,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:50,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f2e2c22[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:50,599 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:50,599 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,599 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,599 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:50,599 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:50,599 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,600 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:50,600 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:50,601 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,601 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:50,601 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:50,601 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:50,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:50,603 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:50,603 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:50,603 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:50,603 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:50,603 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:50,604 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,604 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:50,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:50,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,604 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:50,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:50,605 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:50,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:50,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:50,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,605 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,607 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:50,607 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:50,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:50,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:50,607 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:50,607 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:50,607 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:50,607 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:50,607 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:50,607 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,607 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,607 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,607 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,607 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:50,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:50,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273930607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:50,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:50,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273930608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273930608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:50,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273930608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,702 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032040d676774d964e58941f1d08f2bd7dd0 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032040d676774d964e58941f1d08f2bd7dd0 2023-03-20 00:57:50,707 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:50,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:50,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:50,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:50,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:50,710 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:50,710 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:50,710 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:50,710 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:50,711 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,711 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,712 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,712 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:50,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:50,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:50,712 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,712 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:50,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:50,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:50,714 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:50,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:50,714 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:50,714 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:50,714 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:50,714 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:50,714 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:50,714 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:50,714 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:50,714 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:50,715 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:50,715 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:50,715 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:50,715 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,715 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:50,715 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:50,716 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,716 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:50,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:50,716 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,716 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:50,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:50,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:50,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:50,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:50,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b8abd09[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:50,717 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:50,717 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,718 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,718 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:50,718 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,718 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:50,718 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:50,718 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:50,719 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,719 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:50,719 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:50,719 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:50,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:50,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:50,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:50,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:50,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:50,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:50,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:50,721 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:50,721 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:50,721 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:50,721 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:50,721 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:50,722 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:50,722 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,722 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:50,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,723 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:50,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,724 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:50,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,727 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,727 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:50,727 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,727 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:50,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:50,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:50,727 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,727 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,727 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,727 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,727 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:50,727 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:50,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:50,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:50,728 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:50,728 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:50,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:50,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:50,828 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:50,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:50,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:50,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:50,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:50,831 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:50,831 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:50,831 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:50,831 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:50,831 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,831 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,832 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,832 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:50,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:50,832 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,832 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:50,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:50,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,833 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:50,833 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:50,833 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:50,833 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:50,833 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:50,833 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:50,833 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:50,833 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:50,833 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:50,833 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:50,833 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:50,833 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:50,833 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:50,833 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:50,833 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,833 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:50,833 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:50,834 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,834 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:50,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:50,834 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,834 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:50,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:50,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:50,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7129d230[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:50,835 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:50,835 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,836 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,836 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,836 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:50,836 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:50,836 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:50,836 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:50,837 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,837 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:50,837 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:50,837 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:50,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:50,838 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:50,839 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:50,839 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:50,839 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:50,839 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:50,839 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,839 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:50,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,840 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:50,840 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,841 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,842 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:50,842 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:50,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:50,842 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:50,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:50,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:50,842 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:50,842 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:50,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:50,842 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,842 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:50,842 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,842 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:50,842 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,843 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,843 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:50,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273930911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:50,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273930911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:50,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273930912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:50,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273930913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,943 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:50,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:50,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:50,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:50,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:50,945 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:50,945 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:50,945 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:50,945 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:50,946 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,946 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,946 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,946 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:50,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:50,947 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,947 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,947 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:50,948 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:50,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:50,948 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:50,948 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:50,948 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:50,948 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:50,948 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:50,948 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:50,948 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:50,948 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:50,948 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:50,948 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:50,948 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:50,948 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,948 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:50,948 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:50,949 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,949 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:50,949 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,949 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:50,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:50,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33fa8fd2[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:50,950 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:50,950 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,951 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,951 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,951 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:50,951 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:50,951 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:50,951 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:50,952 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,952 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:50,952 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:50,952 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:50,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:50,953 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:50,953 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:50,953 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:50,953 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:50,953 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:50,955 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,955 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:50,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,955 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:50,956 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:50,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,957 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:50,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,958 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,958 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:50,958 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,958 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:50,958 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:50,958 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,958 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:50,958 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:50,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:50,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:50,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:50,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:50,958 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:50,958 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:50,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:50,958 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:50,959 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:50,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:50,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:51,059 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:51,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:51,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:51,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:51,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:51,061 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:51,061 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:51,061 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:51,062 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:51,062 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,062 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:51,062 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:51,062 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:51,063 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,063 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:51,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:51,063 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:51,064 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:51,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:51,064 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:51,064 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:51,064 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:51,064 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:51,064 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:51,064 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:51,064 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:51,064 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:51,064 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:51,064 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:51,064 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:51,064 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,064 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:51,064 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:51,065 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,065 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:51,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:51,065 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,065 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:51,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:51,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:51,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dfbfab3[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:51,067 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:51,067 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,067 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,067 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:51,067 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:51,067 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:51,067 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,067 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:51,068 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,068 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:51,068 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:51,068 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:51,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:51,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:51,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:51,071 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:51,071 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:51,071 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:51,071 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:51,071 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:51,091 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:51,092 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,092 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:51,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,092 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:51,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:51,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:51,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:51,095 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:51,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:51,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:51,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,098 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:51,098 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:51,098 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:51,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:51,098 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:51,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:51,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:51,099 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:51,099 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,099 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,099 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:51,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:51,099 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:51,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:51,099 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:51,099 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:51,099 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,109 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=65, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/a338c35e8eca47eaa7bac41675cabf3a 2023-03-20 00:57:51,199 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:51,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:51,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:51,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:51,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:51,202 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:51,202 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:51,202 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:51,202 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:51,203 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,203 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:51,203 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,203 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:51,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:51,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:51,203 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,203 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:51,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:51,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:51,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:51,204 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:51,204 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:51,204 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:51,204 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:51,204 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:51,204 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:51,204 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:51,205 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:51,205 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:51,205 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:51,205 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:51,205 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:51,205 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:51,205 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:51,205 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,205 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:51,205 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:51,206 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,206 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:51,206 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,206 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:51,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:51,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36762f0[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:51,207 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:51,207 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,207 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,207 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:51,207 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:51,207 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,208 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:51,207 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:51,208 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,208 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:51,208 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:51,208 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:51,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:51,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:51,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:51,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:51,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:51,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:51,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:51,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:51,210 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:51,210 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:51,210 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:51,210 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:51,210 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:51,211 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,211 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:51,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,211 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:51,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:51,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:51,212 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:51,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:51,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:51,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:51,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:51,213 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:51,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:51,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,214 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,214 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:51,214 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:51,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:51,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:51,214 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:51,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:51,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:51,214 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:51,214 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:51,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:51,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:51,214 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:51,214 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:51,214 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:51,215 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,215 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,215 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,315 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:51,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:51,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:51,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:51,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:51,317 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:51,317 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:51,317 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:51,317 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:51,318 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,318 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:51,318 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:51,318 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:51,318 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,319 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:51,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:51,319 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:51,319 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:51,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:51,319 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:51,319 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:51,319 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:51,319 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:51,320 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:51,320 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:51,320 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:51,320 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:51,320 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:51,320 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:51,320 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:51,320 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,320 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:51,320 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:51,321 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,321 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:51,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:51,321 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,321 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:51,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:51,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:51,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:51,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:51,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:51,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ce098fe[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:51,322 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:51,322 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,323 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,323 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:51,323 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:51,323 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:51,323 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,323 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:51,323 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:51,324 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:51,323 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,324 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:51,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:51,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:51,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:51,325 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:51,325 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:51,325 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:51,325 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:51,325 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:51,326 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:51,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:51,326 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,326 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:51,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,327 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,328 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,330 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:51,330 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:51,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:51,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:51,330 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,330 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:51,330 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:51,330 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,330 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,330 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:51,330 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:51,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:51,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:51,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:51,330 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,331 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:51,331 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:51,415 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:51,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273931415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,416 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:51,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273931416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:51,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273931416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:51,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273931417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,431 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:51,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:51,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:51,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:51,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:51,433 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:51,433 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:51,433 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:51,433 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:51,434 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,434 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:51,434 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,434 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:51,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:51,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:51,434 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,435 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:51,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:51,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:51,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:51,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:51,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:51,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:51,435 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:51,436 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:51,436 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:51,436 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:51,436 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:51,436 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:51,436 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:51,436 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:51,436 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:51,436 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:51,436 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:51,436 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,436 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:51,436 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:51,437 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,437 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:51,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:51,437 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,437 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:51,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:51,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:51,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:51,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b8314af[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:51,438 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:51,438 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,439 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,439 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:51,439 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:51,439 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:51,439 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,439 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:51,440 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,440 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:51,440 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:51,440 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:51,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:51,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:51,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:51,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:51,442 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:51,442 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:51,442 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:51,442 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:51,442 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:51,442 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:51,443 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,443 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:51,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,443 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:51,444 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:51,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,445 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,445 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:51,445 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:51,445 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:51,445 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,445 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:51,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:51,445 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:51,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:51,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:51,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:51,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:51,445 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,446 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:51,446 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,446 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:51,446 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:51,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:51,546 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:51,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:51,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:51,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:51,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:51,550 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:51,550 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:51,550 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:51,550 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:51,550 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,550 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:51,557 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:51,557 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:51,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:51,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:51,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,558 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,558 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:51,558 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:51,567 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:51,567 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:51,567 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:51,567 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:51,567 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:51,570 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:51,570 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:51,570 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/737986ec6d11401d9b89fd8fc078f805 2023-03-20 00:57:51,570 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:51,571 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:51,571 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:51,571 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:51,571 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:51,571 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:51,571 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,571 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:51,571 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:51,581 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:51,581 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:51,582 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:51,582 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:51,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:51,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:51,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:51,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:51,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@790f530e[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:51,585 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:51,585 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,589 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,589 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:51,589 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:51,589 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:51,590 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:51,590 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:51,590 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:51,591 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,591 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:51,591 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:51,591 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:51,591 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:51,592 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:51,592 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:51,592 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:51,593 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,593 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,593 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:51,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:51,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:51,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,594 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:51,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:51,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:51,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:51,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:51,596 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:51,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:51,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:51,599 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/0602bb8f68e24aba88bad0b1c9f0e1f3 2023-03-20 00:57:51,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,604 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/a338c35e8eca47eaa7bac41675cabf3a as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/a338c35e8eca47eaa7bac41675cabf3a 2023-03-20 00:57:51,606 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:51,606 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:51,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:51,606 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:51,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:51,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:51,606 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:51,606 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:51,606 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:51,606 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:51,606 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,606 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,606 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:51,606 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,606 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:51,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:51,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:51,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:51,616 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/a338c35e8eca47eaa7bac41675cabf3a, entries=150, sequenceid=65, filesize=25.1 K 2023-03-20 00:57:51,616 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/737986ec6d11401d9b89fd8fc078f805 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/737986ec6d11401d9b89fd8fc078f805 2023-03-20 00:57:51,633 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/737986ec6d11401d9b89fd8fc078f805, entries=150, sequenceid=65, filesize=11.6 K 2023-03-20 00:57:51,634 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/0602bb8f68e24aba88bad0b1c9f0e1f3 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/0602bb8f68e24aba88bad0b1c9f0e1f3 2023-03-20 00:57:51,638 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/0602bb8f68e24aba88bad0b1c9f0e1f3, entries=150, sequenceid=65, filesize=11.6 K 2023-03-20 00:57:51,638 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 3796a7d329c46cabe9e95c4c1fb73632 in 1347ms, sequenceid=65, compaction requested=true 2023-03-20 00:57:51,638 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:57:51,638 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:51,639 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:51,639 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:57:51,639 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:57:51,639 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:51,639 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77238 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:57:51,639 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:57:51,639 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 3796a7d329c46cabe9e95c4c1fb73632/A is initiating minor compaction (all files) 2023-03-20 00:57:51,639 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 3796a7d329c46cabe9e95c4c1fb73632/B is initiating minor compaction (all files) 2023-03-20 00:57:51,640 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3796a7d329c46cabe9e95c4c1fb73632/A in TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:51,640 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3796a7d329c46cabe9e95c4c1fb73632/B in TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:51,640 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/606efe733faf45cf99fefcc8449533fa, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/930d42d43f8045c7b8c7594b5bb43859, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/a338c35e8eca47eaa7bac41675cabf3a] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp, totalSize=75.4 K 2023-03-20 00:57:51,640 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/3900c55c86984042ae1c9b7ace407527, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/46c0d7ada6bb4f9d81455536b8fe5a75, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/737986ec6d11401d9b89fd8fc078f805] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp, totalSize=34.8 K 2023-03-20 00:57:51,640 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 606efe733faf45cf99fefcc8449533fa, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1679273868034 2023-03-20 00:57:51,640 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 3900c55c86984042ae1c9b7ace407527, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1679273868034 2023-03-20 00:57:51,640 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 930d42d43f8045c7b8c7594b5bb43859, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679273868054 2023-03-20 00:57:51,640 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 46c0d7ada6bb4f9d81455536b8fe5a75, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679273868054 2023-03-20 00:57:51,640 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting a338c35e8eca47eaa7bac41675cabf3a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1679273870185 2023-03-20 00:57:51,640 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 737986ec6d11401d9b89fd8fc078f805, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1679273870185 2023-03-20 00:57:51,663 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 3796a7d329c46cabe9e95c4c1fb73632#B#compaction#439 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:51,666 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3796a7d329c46cabe9e95c4c1fb73632#A#compaction#438 average throughput is 1.23 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:51,699 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/eaa49ece820b4bcfbeb9833fb1469718 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/eaa49ece820b4bcfbeb9833fb1469718 2023-03-20 00:57:51,703 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3796a7d329c46cabe9e95c4c1fb73632/B of 3796a7d329c46cabe9e95c4c1fb73632 into eaa49ece820b4bcfbeb9833fb1469718(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:51,703 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:57:51,703 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632., storeName=3796a7d329c46cabe9e95c4c1fb73632/B, priority=13, startTime=1679273871639; duration=0sec 2023-03-20 00:57:51,703 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:51,703 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:57:51,704 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:57:51,704 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 3796a7d329c46cabe9e95c4c1fb73632/C is initiating minor compaction (all files) 2023-03-20 00:57:51,704 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3796a7d329c46cabe9e95c4c1fb73632/C in TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:51,704 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/04b5de4c1b2e40749030f64ed7e0ba39, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/b787a30d2a344679a3f6a8b89d5075cb, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/0602bb8f68e24aba88bad0b1c9f0e1f3] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp, totalSize=34.8 K 2023-03-20 00:57:51,704 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 04b5de4c1b2e40749030f64ed7e0ba39, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1679273868034 2023-03-20 00:57:51,704 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting b787a30d2a344679a3f6a8b89d5075cb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679273868054 2023-03-20 00:57:51,705 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 0602bb8f68e24aba88bad0b1c9f0e1f3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1679273870185 2023-03-20 00:57:51,707 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:51,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:51,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:51,709 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 3796a7d329c46cabe9e95c4c1fb73632#C#compaction#440 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:51,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:51,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:51,710 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:51,710 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:51,710 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:51,710 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:51,710 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,710 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:51,711 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:51,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:51,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:51,711 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:51,711 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:51,711 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:51,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:51,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:51,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:51,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:51,722 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:51,722 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:51,722 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:51,734 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:51,734 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:51,734 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:51,734 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:51,734 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3796a7d329c46cabe9e95c4c1fb73632 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-20 00:57:51,734 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=A 2023-03-20 00:57:51,734 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:51,735 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=B 2023-03-20 00:57:51,735 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:51,735 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=C 2023-03-20 00:57:51,735 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:51,737 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/6664106bca4548f2b3926ca0f1844b4c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/6664106bca4548f2b3926ca0f1844b4c 2023-03-20 00:57:51,741 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3796a7d329c46cabe9e95c4c1fb73632/C of 3796a7d329c46cabe9e95c4c1fb73632 into 6664106bca4548f2b3926ca0f1844b4c(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:51,741 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:57:51,741 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632., storeName=3796a7d329c46cabe9e95c4c1fb73632/C, priority=13, startTime=1679273871639; duration=0sec 2023-03-20 00:57:51,741 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:52,159 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320feb82ee0544a492798a0a3290bf0d265 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320feb82ee0544a492798a0a3290bf0d265 2023-03-20 00:57:52,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:52,205 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:52,231 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:52,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49504 deadline: 1679273932230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:52,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:52,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49504 deadline: 1679273932332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:52,419 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:52,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273932417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:52,419 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:52,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273932417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:52,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:52,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273932421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:52,424 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:52,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273932422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:52,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:52,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49504 deadline: 1679273932536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:52,563 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=83, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/a2fa67cc064a46479dd1e32e47148b93 2023-03-20 00:57:52,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:52,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49504 deadline: 1679273932839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:52,896 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/039fe2158935405da3d7dbca3b311a23 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/039fe2158935405da3d7dbca3b311a23 2023-03-20 00:57:52,900 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3796a7d329c46cabe9e95c4c1fb73632/A of 3796a7d329c46cabe9e95c4c1fb73632 into 039fe2158935405da3d7dbca3b311a23(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-20 00:57:52,900 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:57:52,900 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632., storeName=3796a7d329c46cabe9e95c4c1fb73632/A, priority=13, startTime=1679273871638; duration=1sec 2023-03-20 00:57:52,900 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:52,981 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/191ed921d07b4d029528a1eddde6c05d 2023-03-20 00:57:53,005 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/9b6989ea5c6041f9b3fa766883c7070e 2023-03-20 00:57:53,010 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/a2fa67cc064a46479dd1e32e47148b93 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/a2fa67cc064a46479dd1e32e47148b93 2023-03-20 00:57:53,014 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/a2fa67cc064a46479dd1e32e47148b93, entries=150, sequenceid=83, filesize=25.1 K 2023-03-20 00:57:53,015 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/191ed921d07b4d029528a1eddde6c05d as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/191ed921d07b4d029528a1eddde6c05d 2023-03-20 00:57:53,025 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/191ed921d07b4d029528a1eddde6c05d, entries=150, sequenceid=83, filesize=11.6 K 2023-03-20 00:57:53,026 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/9b6989ea5c6041f9b3fa766883c7070e as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/9b6989ea5c6041f9b3fa766883c7070e 2023-03-20 00:57:53,032 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/9b6989ea5c6041f9b3fa766883c7070e, entries=150, sequenceid=83, filesize=11.6 K 2023-03-20 00:57:53,033 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 3796a7d329c46cabe9e95c4c1fb73632 in 1299ms, sequenceid=83, compaction requested=false 2023-03-20 00:57:53,033 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:57:53,033 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:53,033 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:53,033 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:53,033 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:53,033 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:53,033 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:53,034 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:53,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:53,034 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:53,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:53,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:53,034 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:53,034 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:53,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:53,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:53,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:53,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:53,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:53,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:53,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41c7de3d[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:53,035 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:53,036 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:53,036 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:53,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:53,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:53,036 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:53,036 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:53,036 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:53,036 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:53,036 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:53,037 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:53,037 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:53,037 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:53,037 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:53,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:53,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:53,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:53,039 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:53,039 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:53,039 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:53,039 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:53,039 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:53,040 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:53,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:53,040 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:53,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:53,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:53,040 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:53,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:53,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:53,040 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:53,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:53,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:53,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:53,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:53,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:53,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:53,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:53,042 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:53,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:53,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:53,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:53,045 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:53,045 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:53,045 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:53,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:53,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:53,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:53,045 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:53,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:53,045 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:53,046 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:53,046 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:53,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:53,046 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:53,046 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:53,046 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:53,046 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:53,046 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:53,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:53,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:53,146 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:53,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:53,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:53,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:53,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:53,148 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:53,148 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:53,148 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:53,148 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:53,149 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:53,149 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:53,149 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:53,149 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:53,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:53,149 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:53,149 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:53,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:53,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:53,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:53,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:53,150 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:53,150 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:53,150 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:53,150 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:53,150 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:53,150 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:53,150 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:53,151 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:53,151 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:53,151 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:53,151 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3796a7d329c46cabe9e95c4c1fb73632 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-20 00:57:53,151 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=A 2023-03-20 00:57:53,151 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:53,151 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=B 2023-03-20 00:57:53,151 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:53,151 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=C 2023-03-20 00:57:53,151 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:53,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:53,344 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:53,379 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:53,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49504 deadline: 1679273933377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:53,481 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:53,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49504 deadline: 1679273933480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:53,565 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303201a24d452d9614a85bd782b9f0bd391b5 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303201a24d452d9614a85bd782b9f0bd391b5 2023-03-20 00:57:53,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:53,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49504 deadline: 1679273933683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:53,969 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=100, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/b2deaba938f341389aa0dd3adfc87118 2023-03-20 00:57:53,977 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/a6573eb527d0450abb9e4d226fad778f 2023-03-20 00:57:53,984 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/50a056a76cf7426d8b56ec16ec817966 2023-03-20 00:57:53,987 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/b2deaba938f341389aa0dd3adfc87118 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/b2deaba938f341389aa0dd3adfc87118 2023-03-20 00:57:53,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:53,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49504 deadline: 1679273933987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:53,991 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/b2deaba938f341389aa0dd3adfc87118, entries=150, sequenceid=100, filesize=25.1 K 2023-03-20 00:57:53,991 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/a6573eb527d0450abb9e4d226fad778f as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/a6573eb527d0450abb9e4d226fad778f 2023-03-20 00:57:53,994 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/a6573eb527d0450abb9e4d226fad778f, entries=150, sequenceid=100, filesize=11.6 K 2023-03-20 00:57:53,994 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/50a056a76cf7426d8b56ec16ec817966 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/50a056a76cf7426d8b56ec16ec817966 2023-03-20 00:57:53,996 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/50a056a76cf7426d8b56ec16ec817966, entries=150, sequenceid=100, filesize=11.6 K 2023-03-20 00:57:53,997 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 3796a7d329c46cabe9e95c4c1fb73632 in 846ms, sequenceid=100, compaction requested=true 2023-03-20 00:57:53,997 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:57:53,997 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:53,997 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:53,997 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:53,997 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:53,997 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:53,997 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:53,998 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:53,998 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:53,998 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:53,998 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:53,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:53,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79589993[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:53,999 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:53,999 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:54,000 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:54,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:54,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:54,000 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:54,000 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:54,000 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:54,000 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,000 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:54,001 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:54,001 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,001 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:54,001 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:54,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:54,002 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:54,003 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:54,003 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:54,003 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:54,003 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:54,003 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,003 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,003 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:54,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:54,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:54,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:54,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:54,004 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:54,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:54,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:54,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:54,005 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:54,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:54,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,006 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:54,006 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:54,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:54,006 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:54,006 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:54,006 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:54,006 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:54,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:54,007 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:54,006 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:54,006 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:54,007 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:54,007 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:54,007 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:54,107 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:54,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:54,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:54,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:54,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:54,110 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:54,110 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:54,110 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:54,110 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:54,110 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,110 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:54,111 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:54,111 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:54,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:54,112 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,112 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:54,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:54,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:54,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:54,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:54,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:54,118 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:54,118 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:54,118 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:54,119 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:54,119 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:54,119 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:54,119 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:54,119 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3796a7d329c46cabe9e95c4c1fb73632 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-20 00:57:54,120 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=A 2023-03-20 00:57:54,120 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:54,120 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=B 2023-03-20 00:57:54,120 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:54,120 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=C 2023-03-20 00:57:54,120 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:54,135 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320e50f299287004606852580a66af9dd3c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320e50f299287004606852580a66af9dd3c 2023-03-20 00:57:54,184 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=117, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/3840728968a048a48551609b8faefa54 2023-03-20 00:57:54,252 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/a16304ef2a28409a8687e7d75b4d9fc0 2023-03-20 00:57:54,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:54,428 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:54,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:54,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273934439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:54,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273934440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:54,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273934440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:54,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273934441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,495 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:54,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49504 deadline: 1679273934494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:54,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273934542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:54,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273934543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:54,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273934544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:54,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273934544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,680 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/8c104384611e496faf969cc401ee6679 2023-03-20 00:57:54,682 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/3840728968a048a48551609b8faefa54 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/3840728968a048a48551609b8faefa54 2023-03-20 00:57:54,685 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/3840728968a048a48551609b8faefa54, entries=150, sequenceid=117, filesize=25.1 K 2023-03-20 00:57:54,686 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/a16304ef2a28409a8687e7d75b4d9fc0 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/a16304ef2a28409a8687e7d75b4d9fc0 2023-03-20 00:57:54,689 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/a16304ef2a28409a8687e7d75b4d9fc0, entries=150, sequenceid=117, filesize=11.6 K 2023-03-20 00:57:54,689 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/8c104384611e496faf969cc401ee6679 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/8c104384611e496faf969cc401ee6679 2023-03-20 00:57:54,692 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/8c104384611e496faf969cc401ee6679, entries=150, sequenceid=117, filesize=11.6 K 2023-03-20 00:57:54,692 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 3796a7d329c46cabe9e95c4c1fb73632 in 573ms, sequenceid=117, compaction requested=true 2023-03-20 00:57:54,692 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:57:54,692 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:54,693 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:54,693 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:54,693 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,693 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:54,693 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:54,694 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:54,694 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:54,694 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:54,694 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:54,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:54,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:54,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a3dcc7b[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:54,695 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:54,695 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:54,696 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:54,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:54,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:54,696 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:54,696 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,696 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:54,696 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:54,696 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:54,697 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:54,697 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:54,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:54,697 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:54,697 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:54,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:54,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:54,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:54,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:54,699 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:54,699 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:54,699 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:54,699 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:54,699 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:54,699 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,700 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,700 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:54,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:54,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:54,700 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:54,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:54,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:54,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:54,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:54,701 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:54,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:54,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:54,702 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,702 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:54,702 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:54,702 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:54,702 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,702 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:54,702 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:54,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:54,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:54,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:54,702 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:54,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:54,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:54,702 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:54,703 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:54,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:54,703 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:54,703 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:54,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:54,746 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3796a7d329c46cabe9e95c4c1fb73632 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-20 00:57:54,747 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=A 2023-03-20 00:57:54,747 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:54,747 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=B 2023-03-20 00:57:54,747 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:54,747 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=C 2023-03-20 00:57:54,747 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:54,758 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320ed43d0feb09240c09f92507bf7094480 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320ed43d0feb09240c09f92507bf7094480 2023-03-20 00:57:54,759 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:54,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273934755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:54,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273934757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,761 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:54,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273934758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,761 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:54,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273934759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,762 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=134, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/90245c25b766488c90e61b9bb56ce5ee 2023-03-20 00:57:54,803 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:54,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:54,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:54,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:54,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:54,806 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:54,806 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:54,806 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:54,806 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:54,806 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,806 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:54,807 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:54,807 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:54,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:54,807 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,807 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:54,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:54,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:54,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:54,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:54,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:54,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:54,808 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:54,808 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:54,808 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:54,808 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:54,808 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:54,808 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:54,808 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:54,808 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:54,808 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:54,808 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:54,808 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:54,808 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,808 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:54,808 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:54,809 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,809 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:54,809 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:54,809 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:54,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:54,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39296f11[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:54,810 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:54,810 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:54,811 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:54,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:54,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:54,811 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:54,811 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:54,811 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,811 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:54,811 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:54,812 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,812 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:54,812 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:54,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:54,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:54,812 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:54,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:54,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:54,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:54,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:54,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:54,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:54,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:54,813 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:54,814 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:54,814 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:54,814 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:54,814 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:54,814 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,814 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:54,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:54,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,814 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,815 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:54,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:54,816 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:54,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:54,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,817 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,817 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:54,817 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:54,817 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:54,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:54,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:54,817 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:54,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:54,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:54,817 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:54,817 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,818 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:54,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:54,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:54,818 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:54,818 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:54,818 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:54,818 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:54,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273934860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:54,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273934861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:54,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273934861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,864 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:54,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273934862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,918 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:54,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:54,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:54,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:54,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:54,920 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:54,920 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:54,920 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:54,920 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:54,921 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,921 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:54,921 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:54,921 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:54,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:54,922 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,922 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:54,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:54,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:54,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:54,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:54,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:54,923 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:54,923 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:54,923 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:54,923 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:54,926 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:54,926 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:54,926 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:54,926 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:54,926 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:54,926 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:54,926 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:54,926 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:54,926 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,926 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:54,926 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:54,934 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,934 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:54,934 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:54,934 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:54,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:54,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4660683f[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:54,936 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:54,936 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:54,939 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:54,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:54,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:54,939 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:54,939 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,939 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:54,939 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:54,939 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:54,940 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,940 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:54,940 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:54,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:54,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:54,940 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:54,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:54,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:54,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:54,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:54,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:54,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:54,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:54,942 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:54,942 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:54,942 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:54,942 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:54,942 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:54,943 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,943 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:54,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:54,943 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:54,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,943 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:54,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:54,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:54,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:54,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:54,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:54,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:54,944 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:54,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:54,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,946 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:54,946 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:54,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:54,946 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:54,946 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:54,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:54,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:54,946 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:54,946 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:54,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:54,946 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:54,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:54,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:54,946 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:54,946 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:54,946 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:54,946 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:54,946 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,046 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:55,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:55,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:55,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:55,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:55,049 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:55,049 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:55,049 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:55,049 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:55,049 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,050 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,051 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,051 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:55,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:55,051 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,051 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:55,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,052 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:55,052 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:55,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:55,052 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:55,052 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:55,052 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:55,058 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:55,058 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:55,058 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:55,058 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:55,058 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:55,058 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:55,058 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:55,058 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:55,058 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,058 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:55,058 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:55,064 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,064 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:55,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:55,064 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273935063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,064 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:55,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:55,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273935063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:55,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273935064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:55,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:55,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:55,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:55,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bc9f561[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:55,066 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:55,066 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,067 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,067 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:55,067 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:55,067 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:55,067 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,067 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:55,067 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,067 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:55,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:55,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:55,068 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:55,068 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:55,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:55,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:55,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273935065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:55,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:55,070 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:55,070 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:55,070 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:55,070 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:55,070 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:55,071 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:55,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:55,071 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,071 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:55,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:55,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:55,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:55,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,072 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:55,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,073 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:55,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,074 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,074 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,074 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:55,074 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:55,074 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:55,074 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:55,074 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:55,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:55,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:55,074 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:55,075 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,075 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,075 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:55,075 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:55,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:55,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:55,169 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/3d3bb2215bdf44ae8fcc516c9fabd718 2023-03-20 00:57:55,175 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:55,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:55,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:55,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:55,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:55,186 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:55,186 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:55,186 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:55,186 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:55,187 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,187 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,187 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,187 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:55,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:55,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:55,188 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,188 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:55,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:55,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,188 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:55,194 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:55,194 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:55,194 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:55,194 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:55,194 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:55,194 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:55,194 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:55,194 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:55,194 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:55,194 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:55,194 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:55,195 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:55,195 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:55,195 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,195 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:55,195 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:55,195 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/53b146ebebaf4e7696f9e50f5c9b36b5 2023-03-20 00:57:55,197 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:55,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:55,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:55,197 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:55,198 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,198 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:55,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,199 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/90245c25b766488c90e61b9bb56ce5ee as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/90245c25b766488c90e61b9bb56ce5ee 2023-03-20 00:57:55,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:55,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:55,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fc3b077[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:55,199 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:55,199 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,200 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,200 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:55,200 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:55,200 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:55,201 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,201 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:55,201 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:55,201 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:55,201 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:55,201 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:55,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:55,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:55,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,203 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/90245c25b766488c90e61b9bb56ce5ee, entries=150, sequenceid=134, filesize=25.3 K 2023-03-20 00:57:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,204 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/3d3bb2215bdf44ae8fcc516c9fabd718 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/3d3bb2215bdf44ae8fcc516c9fabd718 2023-03-20 00:57:55,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:55,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:55,204 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:55,204 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:55,204 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:55,204 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:55,204 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:55,210 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/3d3bb2215bdf44ae8fcc516c9fabd718, entries=150, sequenceid=134, filesize=11.8 K 2023-03-20 00:57:55,211 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/53b146ebebaf4e7696f9e50f5c9b36b5 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/53b146ebebaf4e7696f9e50f5c9b36b5 2023-03-20 00:57:55,214 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,214 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,214 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:55,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:55,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:55,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:55,215 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:55,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,216 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/53b146ebebaf4e7696f9e50f5c9b36b5, entries=150, sequenceid=134, filesize=11.8 K 2023-03-20 00:57:55,216 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:55,217 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 3796a7d329c46cabe9e95c4c1fb73632 in 471ms, sequenceid=134, compaction requested=true 2023-03-20 00:57:55,217 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:57:55,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,217 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:55,217 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:55,217 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-20 00:57:55,217 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-20 00:57:55,217 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:55,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,221 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59683 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-20 00:57:55,221 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 3796a7d329c46cabe9e95c4c1fb73632/B is initiating minor compaction (all files) 2023-03-20 00:57:55,221 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3796a7d329c46cabe9e95c4c1fb73632/B in TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:55,221 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/eaa49ece820b4bcfbeb9833fb1469718, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/191ed921d07b4d029528a1eddde6c05d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/a6573eb527d0450abb9e4d226fad778f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/a16304ef2a28409a8687e7d75b4d9fc0, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/3d3bb2215bdf44ae8fcc516c9fabd718] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp, totalSize=58.3 K 2023-03-20 00:57:55,222 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 128983 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-20 00:57:55,222 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting eaa49ece820b4bcfbeb9833fb1469718, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1679273870185 2023-03-20 00:57:55,222 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 3796a7d329c46cabe9e95c4c1fb73632/A is initiating minor compaction (all files) 2023-03-20 00:57:55,222 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3796a7d329c46cabe9e95c4c1fb73632/A in TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:55,222 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/039fe2158935405da3d7dbca3b311a23, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/a2fa67cc064a46479dd1e32e47148b93, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/b2deaba938f341389aa0dd3adfc87118, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/3840728968a048a48551609b8faefa54, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/90245c25b766488c90e61b9bb56ce5ee] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp, totalSize=126.0 K 2023-03-20 00:57:55,222 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 191ed921d07b4d029528a1eddde6c05d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679273870302 2023-03-20 00:57:55,222 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 039fe2158935405da3d7dbca3b311a23, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1679273870185 2023-03-20 00:57:55,222 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting a6573eb527d0450abb9e4d226fad778f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1679273872217 2023-03-20 00:57:55,223 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting a2fa67cc064a46479dd1e32e47148b93, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679273870302 2023-03-20 00:57:55,223 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting a16304ef2a28409a8687e7d75b4d9fc0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1679273873365 2023-03-20 00:57:55,223 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 3d3bb2215bdf44ae8fcc516c9fabd718, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679273874440 2023-03-20 00:57:55,223 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting b2deaba938f341389aa0dd3adfc87118, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1679273872217 2023-03-20 00:57:55,225 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:55,225 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:55,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:55,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:55,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:55,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:55,225 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,225 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:55,226 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,226 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:55,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:55,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:55,226 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,226 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,226 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,226 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,226 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,226 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:55,227 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 3840728968a048a48551609b8faefa54, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1679273873365 2023-03-20 00:57:55,230 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 90245c25b766488c90e61b9bb56ce5ee, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679273874440 2023-03-20 00:57:55,245 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 3796a7d329c46cabe9e95c4c1fb73632#B#compaction#453 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:55,252 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3796a7d329c46cabe9e95c4c1fb73632#A#compaction#454 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:55,262 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/58b88b04c13e4d0297532175f26249fe as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/58b88b04c13e4d0297532175f26249fe 2023-03-20 00:57:55,267 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3796a7d329c46cabe9e95c4c1fb73632/B of 3796a7d329c46cabe9e95c4c1fb73632 into 58b88b04c13e4d0297532175f26249fe(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:55,267 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:57:55,267 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632., storeName=3796a7d329c46cabe9e95c4c1fb73632/B, priority=11, startTime=1679273875217; duration=0sec 2023-03-20 00:57:55,267 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:55,267 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-20 00:57:55,269 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59683 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-20 00:57:55,269 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 3796a7d329c46cabe9e95c4c1fb73632/C is initiating minor compaction (all files) 2023-03-20 00:57:55,269 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3796a7d329c46cabe9e95c4c1fb73632/C in TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:55,269 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/6664106bca4548f2b3926ca0f1844b4c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/9b6989ea5c6041f9b3fa766883c7070e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/50a056a76cf7426d8b56ec16ec817966, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/8c104384611e496faf969cc401ee6679, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/53b146ebebaf4e7696f9e50f5c9b36b5] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp, totalSize=58.3 K 2023-03-20 00:57:55,269 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 6664106bca4548f2b3926ca0f1844b4c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1679273870185 2023-03-20 00:57:55,270 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 9b6989ea5c6041f9b3fa766883c7070e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679273870302 2023-03-20 00:57:55,270 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 50a056a76cf7426d8b56ec16ec817966, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1679273872217 2023-03-20 00:57:55,270 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 8c104384611e496faf969cc401ee6679, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1679273873365 2023-03-20 00:57:55,271 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 53b146ebebaf4e7696f9e50f5c9b36b5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679273874440 2023-03-20 00:57:55,279 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 3796a7d329c46cabe9e95c4c1fb73632#C#compaction#455 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:55,287 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/589b0ba7b3914b5e8545f75474ce2fd4 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/589b0ba7b3914b5e8545f75474ce2fd4 2023-03-20 00:57:55,290 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3796a7d329c46cabe9e95c4c1fb73632/C of 3796a7d329c46cabe9e95c4c1fb73632 into 589b0ba7b3914b5e8545f75474ce2fd4(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:55,290 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:57:55,290 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632., storeName=3796a7d329c46cabe9e95c4c1fb73632/C, priority=11, startTime=1679273875217; duration=0sec 2023-03-20 00:57:55,290 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:55,326 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:55,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:55,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:55,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:55,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:55,332 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:55,332 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:55,332 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:55,332 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:55,333 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,333 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,333 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:55,333 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:55,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:55,334 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,334 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:55,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,334 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:55,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,334 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:55,335 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:55,335 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:55,335 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:55,335 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:55,335 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:55,335 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:55,335 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:55,335 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:55,335 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:55,335 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3796a7d329c46cabe9e95c4c1fb73632 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-20 00:57:55,335 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=A 2023-03-20 00:57:55,335 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:55,335 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=B 2023-03-20 00:57:55,335 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:55,335 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=C 2023-03-20 00:57:55,335 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:55,356 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032037584bd6cccf430db0a603b95e0e3d17 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032037584bd6cccf430db0a603b95e0e3d17 2023-03-20 00:57:55,360 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=151, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/0a6edebdbc0449f897117f8d8d7e0d14 2023-03-20 00:57:55,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:55,367 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:55,369 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=151 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/53c7f5aca1c140348d2c3d4b1189e0cc 2023-03-20 00:57:55,378 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=151 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/0ab92f8a8ed34e2690a032c45b6b8b96 2023-03-20 00:57:55,381 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/0a6edebdbc0449f897117f8d8d7e0d14 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/0a6edebdbc0449f897117f8d8d7e0d14 2023-03-20 00:57:55,384 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/0a6edebdbc0449f897117f8d8d7e0d14, entries=150, sequenceid=151, filesize=25.3 K 2023-03-20 00:57:55,384 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/53c7f5aca1c140348d2c3d4b1189e0cc as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/53c7f5aca1c140348d2c3d4b1189e0cc 2023-03-20 00:57:55,391 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/53c7f5aca1c140348d2c3d4b1189e0cc, entries=150, sequenceid=151, filesize=11.8 K 2023-03-20 00:57:55,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:55,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273935384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:55,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273935384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:55,391 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/0ab92f8a8ed34e2690a032c45b6b8b96 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/0ab92f8a8ed34e2690a032c45b6b8b96 2023-03-20 00:57:55,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273935386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,392 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:55,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273935388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,405 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/0ab92f8a8ed34e2690a032c45b6b8b96, entries=150, sequenceid=151, filesize=11.8 K 2023-03-20 00:57:55,406 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 3796a7d329c46cabe9e95c4c1fb73632 in 71ms, sequenceid=151, compaction requested=false 2023-03-20 00:57:55,406 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:57:55,406 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:55,406 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:55,406 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:55,406 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,406 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:55,406 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:55,407 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:55,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:55,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5de00379[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:55,409 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:55,409 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,412 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,412 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:55,421 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,421 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,422 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,422 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:55,422 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:55,422 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:55,422 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:55,425 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,425 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:55,425 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:55,425 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:55,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:55,427 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:55,427 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:55,427 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:55,427 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:55,427 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:55,428 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,428 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:55,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:55,428 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,429 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,431 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,433 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:55,433 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,433 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:55,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:55,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:55,433 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,433 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,433 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,433 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,433 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,433 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:55,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:55,433 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:55,433 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:55,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:55,433 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:55,433 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:55,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:55,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:55,494 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3796a7d329c46cabe9e95c4c1fb73632 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-20 00:57:55,494 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=A 2023-03-20 00:57:55,494 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:55,494 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=B 2023-03-20 00:57:55,494 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:55,494 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=C 2023-03-20 00:57:55,495 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:55,503 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320fa175949f1f24fd9b6babda5bb07e262 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320fa175949f1f24fd9b6babda5bb07e262 2023-03-20 00:57:55,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:55,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273935501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,507 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=170, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/aa0f834a18394985ab007c2a59e7db3a 2023-03-20 00:57:55,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:55,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:55,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49504 deadline: 1679273935502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273935503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,508 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:55,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:55,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273935503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273935503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,533 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:55,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:55,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:55,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:55,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:55,536 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:55,536 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:55,536 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:55,536 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:55,537 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,537 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,538 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:55,538 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:55,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:55,538 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,538 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:55,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:55,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:55,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:55,539 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:55,539 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:55,539 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:55,539 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:55,539 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:55,539 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:55,539 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:55,539 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:55,539 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:55,539 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:55,539 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:55,539 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:55,539 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,539 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:55,539 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:55,540 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,540 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:55,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:55,540 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,540 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:55,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:55,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:55,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:55,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:55,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1884b96a[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:55,541 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:55,541 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,541 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,541 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:55,541 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:55,542 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:55,542 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,542 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:55,542 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,542 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:55,542 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:55,542 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:55,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:55,544 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:55,544 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:55,544 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:55,544 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:55,544 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:55,545 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,545 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:55,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,545 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:55,545 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:55,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:55,547 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:55,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,566 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:55,566 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:55,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:55,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:55,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:55,566 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,566 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,566 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,566 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,566 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,566 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,566 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,566 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:55,566 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:55,567 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:55,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:55,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:55,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:55,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:55,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:55,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273935605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,611 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:55,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273935609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:55,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:55,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273935609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273935609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,667 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:55,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:55,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:55,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:55,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:55,670 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:55,670 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:55,670 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:55,670 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:55,670 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,670 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,671 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:55,671 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:55,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:55,671 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,671 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:55,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:55,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,671 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:55,672 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:55,672 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:55,672 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:55,672 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:55,672 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:55,674 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:55,674 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:55,674 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:55,674 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:55,674 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:55,674 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:55,674 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:55,674 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:55,674 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,674 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:55,674 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:55,675 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,675 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:55,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:55,675 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,675 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:55,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:55,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:55,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:55,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:55,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cd225e6[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:55,676 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:55,676 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,677 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,677 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:55,677 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:55,677 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:55,677 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,677 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:55,678 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,678 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:55,678 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:55,678 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:55,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:55,679 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:55,679 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:55,679 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:55,679 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:55,679 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:55,680 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:55,680 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,680 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:55,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:55,680 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:55,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:55,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,681 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:55,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,683 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,683 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,683 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:55,683 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:55,683 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:55,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:55,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:55,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:55,683 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:55,683 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,683 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:55,683 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,683 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:55,683 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,683 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:55,783 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:55,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:55,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:55,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:55,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:55,786 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:55,786 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:55,786 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:55,786 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:55,786 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,786 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,787 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,787 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:55,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:55,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:55,787 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,787 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:55,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:55,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,788 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:55,788 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:55,788 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:55,788 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:55,788 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:55,788 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:55,788 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:55,788 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:55,788 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:55,788 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:55,788 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:55,789 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:55,789 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:55,789 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:55,789 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,789 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:55,789 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:55,789 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,789 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:55,790 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,790 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:55,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:55,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:55,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:55,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:55,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ea06803[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:55,791 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:55,791 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,792 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,792 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:55,792 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:55,792 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:55,792 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,792 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:55,793 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:55,793 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:55,793 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,793 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:55,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:55,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:55,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:55,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:55,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:55,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:55,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:55,795 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:55,795 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:55,795 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:55,795 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:55,796 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:55,796 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,796 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,796 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:55,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:55,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:55,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:55,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:55,797 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:55,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:55,797 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:55,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,799 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,799 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,799 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:55,799 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:55,799 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:55,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:55,799 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:55,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:55,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:55,799 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:55,799 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,799 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:55,799 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,799 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:55,799 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:55,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:55,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:55,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273935809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:55,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273935813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:55,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273935814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,817 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:55,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273935814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,899 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:55,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:55,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:55,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:55,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:55,902 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:55,902 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:55,902 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:55,902 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:55,902 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,903 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,903 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,903 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:55,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:55,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:55,903 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,903 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:55,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:55,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:55,904 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:55,904 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:55,904 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:55,904 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:55,904 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:55,904 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:55,904 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:55,904 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:55,904 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:55,905 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:55,905 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:55,905 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:55,905 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:55,905 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,905 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:55,905 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:55,905 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,905 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:55,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:55,906 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,906 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:55,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:55,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:55,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:55,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:55,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43065b65[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:55,907 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:55,907 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,907 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,907 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:55,907 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:55,907 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:55,908 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,908 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:55,908 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:55,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:55,908 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:55,908 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:55,908 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:55,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:55,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:55,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:55,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:55,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:55,910 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:55,910 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:55,910 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:55,910 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:55,910 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:55,911 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:55,913 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,913 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:55,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,913 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:55,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,914 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/c6fc8cf8ab5f47f98c005092098c757b 2023-03-20 00:57:55,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:55,914 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:55,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,924 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:55,925 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:55,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:55,925 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:55,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:55,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:55,925 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:55,925 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:55,925 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:55,925 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:55,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:55,925 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:55,925 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:55,925 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,925 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:55,925 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:56,025 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:56,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:56,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:56,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:56,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:56,033 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:56,033 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:56,033 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:56,033 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:56,038 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:56,038 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:56,039 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,039 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:56,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:56,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:56,039 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,039 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:56,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:56,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:56,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:56,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:56,040 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:56,042 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:56,042 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:56,042 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:56,042 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:56,042 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:56,042 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:56,042 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:56,042 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:56,042 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:56,042 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:56,042 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:56,043 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:56,043 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,043 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:56,043 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:56,043 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,043 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:56,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:56,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:56,044 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:56,044 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:56,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:56,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:56,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:56,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:56,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:56,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e74f586[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:56,046 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:56,046 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:56,047 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:56,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:56,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:56,047 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:56,047 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:56,047 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:56,047 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,047 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:56,048 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,048 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:56,048 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:56,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:56,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:56,048 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:56,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:56,054 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:56,054 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:56,054 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:56,054 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:56,054 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:56,066 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:56,066 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:56,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:56,066 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:56,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:56,067 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:56,078 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/3b5671602e0f4e2085a1f33047f9a314 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/3b5671602e0f4e2085a1f33047f9a314 2023-03-20 00:57:56,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:56,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:56,079 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:56,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:56,083 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3796a7d329c46cabe9e95c4c1fb73632/A of 3796a7d329c46cabe9e95c4c1fb73632 into 3b5671602e0f4e2085a1f33047f9a314(size=25.6 K), total size for store is 50.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:56,083 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:57:56,083 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632., storeName=3796a7d329c46cabe9e95c4c1fb73632/A, priority=11, startTime=1679273875217; duration=0sec 2023-03-20 00:57:56,083 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:56,084 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:56,084 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:56,084 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:56,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:56,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:56,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:56,084 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,084 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:56,084 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:56,085 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,085 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:56,085 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:56,085 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:56,085 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:56,085 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:56,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:56,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:56,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:56,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:56,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273936112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,117 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:56,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273936115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:56,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273936118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:56,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273936119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,185 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:56,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:56,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:56,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:56,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:56,188 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:56,188 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:56,188 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:56,188 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:56,189 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:56,189 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:56,189 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,189 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:56,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:56,190 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,190 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:56,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:56,191 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:56,191 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:56,191 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:56,191 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:56,191 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:56,191 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:56,192 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:56,192 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:56,192 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:56,192 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:56,192 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:56,192 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:56,192 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:56,192 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:56,192 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,192 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:56,192 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:56,197 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,197 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:56,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:56,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:56,198 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:56,198 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:56,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:56,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:56,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:56,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:56,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:56,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@433412bb[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:56,199 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:56,199 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:56,200 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:56,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:56,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:56,200 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:56,200 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:56,200 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:56,200 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,200 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:56,201 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:56,201 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:56,201 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:56,201 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:56,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:56,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:56,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:56,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:56,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:56,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:56,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:56,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:56,204 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:56,204 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:56,204 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:56,204 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:56,204 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:56,206 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:56,206 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:56,206 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:56,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:56,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:56,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:56,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:56,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:56,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:56,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:56,207 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:56,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:56,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:56,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:56,208 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:56,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:56,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,210 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,210 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:56,210 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:56,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:56,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:56,210 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,211 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:56,211 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:56,211 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:56,210 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:56,211 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:56,211 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:56,211 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:56,211 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:56,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:56,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:56,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:56,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:56,311 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:56,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:56,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:56,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:56,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:56,314 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:56,314 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:56,314 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:56,314 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:56,314 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:56,314 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:56,315 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:56,315 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:56,315 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:56,315 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:56,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:56,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:56,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:56,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:56,316 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:56,316 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:56,316 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:56,316 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:56,316 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:56,316 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:56,317 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:56,317 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:56,317 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:56,317 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:56,317 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:56,317 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,317 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:56,317 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:56,317 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,317 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:56,318 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:56,318 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:56,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:56,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ec99aae[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:56,319 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:56,319 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:56,320 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:56,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:56,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:56,320 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:56,320 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,320 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:56,320 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:56,320 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:56,321 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,321 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:56,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:56,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:56,321 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:56,321 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:56,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:56,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:56,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:56,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:56,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:56,323 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:56,323 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:56,323 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:56,323 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:56,323 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:56,323 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/823292529b944ad9ba9b5022d9b45102 2023-03-20 00:57:56,324 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:56,324 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:56,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:56,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:56,324 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:56,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:56,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:56,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:56,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:56,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:56,324 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:56,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:56,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:56,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:56,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:56,326 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,327 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/aa0f834a18394985ab007c2a59e7db3a as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/aa0f834a18394985ab007c2a59e7db3a 2023-03-20 00:57:56,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:56,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:56,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,328 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:56,328 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:56,328 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,328 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:56,328 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:56,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:56,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:56,328 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,328 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:56,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:56,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:56,328 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:56,329 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:56,329 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:56,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:56,329 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:56,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:56,329 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:56,330 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/aa0f834a18394985ab007c2a59e7db3a, entries=150, sequenceid=170, filesize=25.3 K 2023-03-20 00:57:56,331 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/c6fc8cf8ab5f47f98c005092098c757b as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/c6fc8cf8ab5f47f98c005092098c757b 2023-03-20 00:57:56,333 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/c6fc8cf8ab5f47f98c005092098c757b, entries=150, sequenceid=170, filesize=11.8 K 2023-03-20 00:57:56,334 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/823292529b944ad9ba9b5022d9b45102 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/823292529b944ad9ba9b5022d9b45102 2023-03-20 00:57:56,336 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/823292529b944ad9ba9b5022d9b45102, entries=150, sequenceid=170, filesize=11.8 K 2023-03-20 00:57:56,337 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 3796a7d329c46cabe9e95c4c1fb73632 in 843ms, sequenceid=170, compaction requested=true 2023-03-20 00:57:56,337 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:57:56,337 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:56,337 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:57:56,337 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:57:56,337 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:56,337 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:56,338 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77962 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:57:56,338 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:57:56,338 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 3796a7d329c46cabe9e95c4c1fb73632/A is initiating minor compaction (all files) 2023-03-20 00:57:56,338 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 3796a7d329c46cabe9e95c4c1fb73632/B is initiating minor compaction (all files) 2023-03-20 00:57:56,338 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3796a7d329c46cabe9e95c4c1fb73632/A in TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:56,338 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3796a7d329c46cabe9e95c4c1fb73632/B in TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:56,338 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/3b5671602e0f4e2085a1f33047f9a314, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/0a6edebdbc0449f897117f8d8d7e0d14, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/aa0f834a18394985ab007c2a59e7db3a] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp, totalSize=76.1 K 2023-03-20 00:57:56,338 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/58b88b04c13e4d0297532175f26249fe, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/53c7f5aca1c140348d2c3d4b1189e0cc, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/c6fc8cf8ab5f47f98c005092098c757b] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp, totalSize=35.5 K 2023-03-20 00:57:56,339 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 3b5671602e0f4e2085a1f33047f9a314, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679273874440 2023-03-20 00:57:56,339 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 58b88b04c13e4d0297532175f26249fe, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679273874440 2023-03-20 00:57:56,339 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 0a6edebdbc0449f897117f8d8d7e0d14, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1679273874754 2023-03-20 00:57:56,339 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 53c7f5aca1c140348d2c3d4b1189e0cc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1679273874754 2023-03-20 00:57:56,339 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting aa0f834a18394985ab007c2a59e7db3a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1679273875373 2023-03-20 00:57:56,339 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting c6fc8cf8ab5f47f98c005092098c757b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1679273875373 2023-03-20 00:57:56,344 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 3796a7d329c46cabe9e95c4c1fb73632#B#compaction#462 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:56,346 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3796a7d329c46cabe9e95c4c1fb73632#A#compaction#463 average throughput is 9.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:56,361 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/1f9ee2f684ea4912809cdffde6c8a42f as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/1f9ee2f684ea4912809cdffde6c8a42f 2023-03-20 00:57:56,365 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3796a7d329c46cabe9e95c4c1fb73632/A of 3796a7d329c46cabe9e95c4c1fb73632 into 1f9ee2f684ea4912809cdffde6c8a42f(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:56,365 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:57:56,365 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632., storeName=3796a7d329c46cabe9e95c4c1fb73632/A, priority=13, startTime=1679273876337; duration=0sec 2023-03-20 00:57:56,365 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:57:56,366 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:57:56,366 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:57:56,366 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 3796a7d329c46cabe9e95c4c1fb73632/C is initiating minor compaction (all files) 2023-03-20 00:57:56,366 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3796a7d329c46cabe9e95c4c1fb73632/C in TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:56,366 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/589b0ba7b3914b5e8545f75474ce2fd4, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/0ab92f8a8ed34e2690a032c45b6b8b96, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/823292529b944ad9ba9b5022d9b45102] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp, totalSize=35.5 K 2023-03-20 00:57:56,367 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 589b0ba7b3914b5e8545f75474ce2fd4, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679273874440 2023-03-20 00:57:56,367 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 0ab92f8a8ed34e2690a032c45b6b8b96, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1679273874754 2023-03-20 00:57:56,367 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 823292529b944ad9ba9b5022d9b45102, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1679273875373 2023-03-20 00:57:56,384 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3796a7d329c46cabe9e95c4c1fb73632#C#compaction#464 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:57:56,391 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/c299cf5972cd4d0fbc4f101e64f882c9 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/c299cf5972cd4d0fbc4f101e64f882c9 2023-03-20 00:57:56,394 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3796a7d329c46cabe9e95c4c1fb73632/C of 3796a7d329c46cabe9e95c4c1fb73632 into c299cf5972cd4d0fbc4f101e64f882c9(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:56,394 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:57:56,394 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632., storeName=3796a7d329c46cabe9e95c4c1fb73632/C, priority=13, startTime=1679273876337; duration=0sec 2023-03-20 00:57:56,395 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:56,429 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:56,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:56,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:56,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:56,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:56,431 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:56,431 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:56,431 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:56,431 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:56,432 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:56,432 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:56,432 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,432 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:56,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:56,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:56,432 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,432 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:56,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:56,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:56,433 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:56,433 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:56,433 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:56,433 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:56,433 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:56,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:56,434 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:56,434 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:56,434 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:56,434 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3796a7d329c46cabe9e95c4c1fb73632 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-20 00:57:56,434 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=A 2023-03-20 00:57:56,434 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:56,434 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=B 2023-03-20 00:57:56,434 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:56,434 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=C 2023-03-20 00:57:56,435 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:56,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:56,619 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:56,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:56,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273936630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,633 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:56,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273936631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,633 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:56,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273936632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:56,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273936633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:56,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273936734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:56,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273936734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:56,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273936734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:56,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273936735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,752 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/aa1ce43eb5fb432f92c3c4ebdc6c96b0 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/aa1ce43eb5fb432f92c3c4ebdc6c96b0 2023-03-20 00:57:56,759 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3796a7d329c46cabe9e95c4c1fb73632/B of 3796a7d329c46cabe9e95c4c1fb73632 into aa1ce43eb5fb432f92c3c4ebdc6c96b0(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:57:56,759 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:57:56,759 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632., storeName=3796a7d329c46cabe9e95c4c1fb73632/B, priority=13, startTime=1679273876337; duration=0sec 2023-03-20 00:57:56,759 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:57:56,850 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032093534997b7524e4a805ed6d9acd2fb39 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032093534997b7524e4a805ed6d9acd2fb39 2023-03-20 00:57:56,860 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=186, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/182bb2d36651472cb12b73a0fc2409ed 2023-03-20 00:57:56,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:56,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273936936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,939 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:56,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273936936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,939 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:56,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273936936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:56,942 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:56,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273936940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:57,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:57,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273937240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:57,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:57,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273937240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:57,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:57,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273937240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:57,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:57,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273937244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:57,277 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/5f5cef1394344482b8b67bc172054b55 2023-03-20 00:57:57,283 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/a49053a121a64085a103be1cbfc02256 2023-03-20 00:57:57,291 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/182bb2d36651472cb12b73a0fc2409ed as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/182bb2d36651472cb12b73a0fc2409ed 2023-03-20 00:57:57,294 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/182bb2d36651472cb12b73a0fc2409ed, entries=150, sequenceid=186, filesize=25.3 K 2023-03-20 00:57:57,295 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/5f5cef1394344482b8b67bc172054b55 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/5f5cef1394344482b8b67bc172054b55 2023-03-20 00:57:57,298 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/5f5cef1394344482b8b67bc172054b55, entries=150, sequenceid=186, filesize=11.8 K 2023-03-20 00:57:57,299 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/a49053a121a64085a103be1cbfc02256 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/a49053a121a64085a103be1cbfc02256 2023-03-20 00:57:57,302 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/a49053a121a64085a103be1cbfc02256, entries=150, sequenceid=186, filesize=11.8 K 2023-03-20 00:57:57,303 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 3796a7d329c46cabe9e95c4c1fb73632 in 869ms, sequenceid=186, compaction requested=false 2023-03-20 00:57:57,303 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:57:57,303 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:57,303 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:57,303 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:57,303 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:57,303 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:57,303 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:57,304 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:57,304 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:57,304 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:57,304 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:57,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:57,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:57,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:57,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:57,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f652f61[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:57,305 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:57,305 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:57,306 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:57,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:57,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:57,306 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:57,306 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:57,306 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:57,306 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:57,306 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:57,307 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:57,307 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:57,307 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:57,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:57,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:57,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:57,307 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:57,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:57,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:57,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:57,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:57,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:57,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:57,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:57,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:57,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:57,309 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:57,309 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:57,309 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:57,309 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:57,309 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:57,310 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:57,310 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:57,310 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:57,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:57,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:57,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:57,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:57,311 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:57,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:57,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:57,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:57,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:57,312 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:57,313 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:57,313 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:57,313 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:57,313 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:57,313 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:57,313 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:57,313 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:57,313 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:57,314 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:57,313 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:57,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:57,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:57,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:57,314 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:57,314 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:57,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:57,414 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:57,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:57,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:57,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:57,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:57,418 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:57,419 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:57,419 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:57,419 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:57,419 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:57,419 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:57,419 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:57,419 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:57,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:57,420 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:57,420 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:57,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:57,421 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:57,422 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:57,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:57,422 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:57,422 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:57,422 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:57,423 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:57,423 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:57,423 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:57,423 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:57,423 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3796a7d329c46cabe9e95c4c1fb73632 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-20 00:57:57,424 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=A 2023-03-20 00:57:57,424 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:57,424 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=B 2023-03-20 00:57:57,424 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:57,424 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=C 2023-03-20 00:57:57,424 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:57,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:57,524 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:57,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:57,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49504 deadline: 1679273937548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:57,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:57,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49504 deadline: 1679273937650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:57,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:57,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273937742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:57,745 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:57,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273937744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:57,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:57,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273937745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:57,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:57,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273937750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:57,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:57,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49504 deadline: 1679273937853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:57,855 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303209699294374a2442d8dd3fb78f83e911d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303209699294374a2442d8dd3fb78f83e911d 2023-03-20 00:57:58,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:58,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49504 deadline: 1679273938155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:58,260 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=206, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/9cc7db9907e14fa3bccbe944df38e00a 2023-03-20 00:57:58,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:58,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49504 deadline: 1679273938657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:58,671 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/85d763ed2a5442389a68ece12bd82042 2023-03-20 00:57:58,750 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:58,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273938749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:58,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:58,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273938751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:58,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:58,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273938753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:58,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:58,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273938757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,080 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/cd716c731fbe41db8e52aaa749112c0c 2023-03-20 00:57:59,083 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/9cc7db9907e14fa3bccbe944df38e00a as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/9cc7db9907e14fa3bccbe944df38e00a 2023-03-20 00:57:59,095 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/9cc7db9907e14fa3bccbe944df38e00a, entries=150, sequenceid=206, filesize=25.3 K 2023-03-20 00:57:59,096 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/85d763ed2a5442389a68ece12bd82042 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/85d763ed2a5442389a68ece12bd82042 2023-03-20 00:57:59,099 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/85d763ed2a5442389a68ece12bd82042, entries=150, sequenceid=206, filesize=11.8 K 2023-03-20 00:57:59,099 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/cd716c731fbe41db8e52aaa749112c0c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/cd716c731fbe41db8e52aaa749112c0c 2023-03-20 00:57:59,103 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/cd716c731fbe41db8e52aaa749112c0c, entries=150, sequenceid=206, filesize=11.8 K 2023-03-20 00:57:59,104 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 3796a7d329c46cabe9e95c4c1fb73632 in 1681ms, sequenceid=206, compaction requested=true 2023-03-20 00:57:59,104 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:57:59,104 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:59,104 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:59,104 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:59,104 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,104 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:59,104 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:59,106 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,106 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:59,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:59,112 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:59,112 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:59,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:59,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:59,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:59,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:59,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@766427ce[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:59,114 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:59,114 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,122 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,122 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:59,122 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:59,122 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:59,123 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,123 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:59,123 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,123 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:59,123 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:59,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:59,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:59,123 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:59,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:59,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:59,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:59,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:59,125 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:59,126 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:59,126 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:59,126 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:59,126 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:59,127 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,127 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:59,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,127 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:59,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:59,127 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,128 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:59,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:59,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,130 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:59,131 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:59,131 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:59,131 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:59,131 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:59,131 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:59,131 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:59,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:59,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:59,131 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,131 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,131 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,131 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:59,131 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:59,231 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:59,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:59,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:59,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:59,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:59,234 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:59,234 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:59,234 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:59,234 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:59,234 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,234 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:59,235 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,236 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:59,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:59,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:59,236 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,236 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:59,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:59,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:59,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:59,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:59,237 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:59,237 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:59,237 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:59,237 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:59,237 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:59,237 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:59,237 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:59,237 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:59,237 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:59,237 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3796a7d329c46cabe9e95c4c1fb73632 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-20 00:57:59,237 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=A 2023-03-20 00:57:59,237 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:59,237 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=B 2023-03-20 00:57:59,237 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:59,238 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=C 2023-03-20 00:57:59,238 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:59,254 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032084a583b4f4c64f2198ba63a7da3b878d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032084a583b4f4c64f2198ba63a7da3b878d 2023-03-20 00:57:59,659 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=219, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/beffb6edf5934c6bb6d05877f9606e6a 2023-03-20 00:57:59,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:59,667 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:59,670 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/588cdb929d494cb783e589e32f07a91f 2023-03-20 00:57:59,693 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/50c4354bdf024447864015a72e2007b6 2023-03-20 00:57:59,697 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/beffb6edf5934c6bb6d05877f9606e6a as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/beffb6edf5934c6bb6d05877f9606e6a 2023-03-20 00:57:59,700 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/beffb6edf5934c6bb6d05877f9606e6a, entries=150, sequenceid=219, filesize=25.3 K 2023-03-20 00:57:59,701 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/588cdb929d494cb783e589e32f07a91f as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/588cdb929d494cb783e589e32f07a91f 2023-03-20 00:57:59,705 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/588cdb929d494cb783e589e32f07a91f, entries=150, sequenceid=219, filesize=11.8 K 2023-03-20 00:57:59,706 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/50c4354bdf024447864015a72e2007b6 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/50c4354bdf024447864015a72e2007b6 2023-03-20 00:57:59,709 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/50c4354bdf024447864015a72e2007b6, entries=150, sequenceid=219, filesize=11.8 K 2023-03-20 00:57:59,710 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=93.93 KB/96180 for 3796a7d329c46cabe9e95c4c1fb73632 in 473ms, sequenceid=219, compaction requested=true 2023-03-20 00:57:59,710 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:57:59,710 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:59,710 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:59,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:57:59,710 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:59,711 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,711 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3796a7d329c46cabe9e95c4c1fb73632 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-20 00:57:59,711 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:59,711 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:59,711 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=A 2023-03-20 00:57:59,711 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:59,711 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=B 2023-03-20 00:57:59,711 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:59,711 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=C 2023-03-20 00:57:59,711 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:57:59,712 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,712 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:59,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:59,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:59,712 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,712 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:59,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:59,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:59,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:59,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:59,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15a64273[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:59,714 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:59,714 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,715 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,715 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:59,715 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:59,715 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:59,715 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,715 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:59,719 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,719 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:59,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:59,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:59,719 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:59,719 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:59,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:59,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:59,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:59,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:59,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:59,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:59,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:59,722 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:59,722 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:59,722 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:59,722 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:59,722 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:59,723 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,723 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:59,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:59,723 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:59,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,723 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032036687c9f7ebb4d0fbc47dc0343242352 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032036687c9f7ebb4d0fbc47dc0343242352 2023-03-20 00:57:59,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:59,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:59,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:59,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:59,725 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:59,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:59,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:59,726 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:59,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,728 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,728 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:59,728 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:59,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:59,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:59,728 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:59,728 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:59,728 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,728 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,728 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,729 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,729 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=237, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/6fb41291e899495ba3e9528292d314d6 2023-03-20 00:57:59,729 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:59,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:59,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:59,729 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:59,729 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:59,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:59,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:59,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:59,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49504 deadline: 1679273939763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,829 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:59,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:59,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:59,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:59,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:59,838 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:59,838 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:59,838 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:59,838 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:59,839 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,839 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:59,839 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,840 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:59,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:59,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:59,840 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,840 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:59,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:59,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:59,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:59,841 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:59,841 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:59,841 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:59,841 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:59,841 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:59,841 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:59,842 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:59,842 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:59,842 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:59,842 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:59,842 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:59,842 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:59,842 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:59,842 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:59,842 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,842 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:59,842 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:59,845 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:59,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:59,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:59,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:59,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:59,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:59,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:59,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@728b683[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:59,846 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:59,846 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,847 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,847 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:59,847 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,848 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,848 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,848 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:59,848 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:59,848 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:59,848 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:59,849 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:59,849 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:59,849 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,849 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:59,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:59,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:59,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:59,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:59,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:59,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:59,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:59,852 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:59,852 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:59,852 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:59,852 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:59,852 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:59,853 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:59,853 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,853 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:59,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:59,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:59,853 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,854 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:59,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:59,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,856 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,856 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:59,856 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:59,856 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:59,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:59,856 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:59,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:59,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:59,856 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,856 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:59,856 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,856 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:59,856 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:59,856 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:59,856 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,856 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,871 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:57:59,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49504 deadline: 1679273939868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,956 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:57:59,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:57:59,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:57:59,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:57:59,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:57:59,961 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:57:59,961 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:59,961 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:57:59,961 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:57:59,961 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,962 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:59,963 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:59,963 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:59,963 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,963 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:57:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:59,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:57:59,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:59,964 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:57:59,964 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:57:59,964 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:57:59,964 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:57:59,964 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:57:59,964 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:57:59,966 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:57:59,966 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:57:59,966 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:59,966 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:57:59,966 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:57:59,966 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:57:59,966 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:57:59,966 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:57:59,966 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,966 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:57:59,966 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:57:59,967 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,967 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:59,967 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,967 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:57:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:59,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:57:59,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b09f20[Count = 0] remaining members to acquire global barrier 2023-03-20 00:57:59,969 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:57:59,969 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,969 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,970 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:57:59,970 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:57:59,970 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:57:59,970 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,970 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:57:59,970 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:57:59,970 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,970 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:59,970 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:57:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:57:59,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:57:59,973 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:57:59,973 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:57:59,973 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:57:59,973 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:57:59,973 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:57:59,974 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,974 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,974 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:57:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:57:59,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:59,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:59,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:57:59,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:57:59,981 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:59,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:57:59,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:59,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,982 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:57:59,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:57:59,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,989 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,989 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:59,989 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:57:59,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:57:59,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:57:59,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:57:59,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:57:59,989 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:57:59,989 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:57:59,989 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:57:59,990 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:57:59,990 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:57:59,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:57:59,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:57:59,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:57:59,990 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,990 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:57:59,990 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:57:59,990 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,074 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:00,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49504 deadline: 1679273940072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,090 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:58:00,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:58:00,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:58:00,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:58:00,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:58:00,093 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:58:00,093 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:00,093 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:58:00,093 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:58:00,093 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,093 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:00,094 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:00,094 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:00,094 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,094 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:58:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:00,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:58:00,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:00,095 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:58:00,095 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:58:00,095 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:58:00,095 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:58:00,095 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:00,095 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:58:00,095 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:58:00,095 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:58:00,095 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:00,095 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:58:00,095 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:58:00,095 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:00,095 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:58:00,095 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:58:00,095 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,095 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:58:00,095 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:58:00,096 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,096 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:00,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:00,097 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,097 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:58:00,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:00,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:00,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:00,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:00,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:58:00,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56bd928f[Count = 0] remaining members to acquire global barrier 2023-03-20 00:58:00,098 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:58:00,098 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,098 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,098 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:58:00,098 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:58:00,098 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:58:00,099 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,099 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:58:00,099 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:58:00,099 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,099 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:00,099 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:58:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:00,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:58:00,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:58:00,101 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:58:00,101 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:58:00,101 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:58:00,101 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:58:00,101 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:58:00,102 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,102 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:00,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:00,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,102 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:00,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:00,102 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:00,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:00,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:00,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:00,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:00,104 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,105 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,106 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:00,106 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:00,106 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:00,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:58:00,106 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:00,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:00,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:00,106 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:00,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:58:00,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:58:00,106 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,106 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,106 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,106 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,106 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:58:00,106 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:58:00,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:00,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:00,176 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/21038d6bd917471a97dbec7f894f70e9 2023-03-20 00:58:00,206 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:58:00,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:58:00,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:58:00,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:58:00,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:58:00,209 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:58:00,209 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:00,209 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:58:00,209 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:58:00,209 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,209 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:00,210 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:00,210 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:00,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:00,210 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,210 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:58:00,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:00,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:58:00,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:00,211 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:58:00,211 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:58:00,211 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:58:00,211 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:58:00,211 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:00,211 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:58:00,211 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:58:00,211 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:58:00,211 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:00,211 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:58:00,212 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:58:00,212 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:00,212 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:58:00,212 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:58:00,212 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,212 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:58:00,212 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:58:00,212 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:00,213 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:00,213 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:00,213 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:58:00,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:00,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:00,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:00,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:58:00,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@712ff7f1[Count = 0] remaining members to acquire global barrier 2023-03-20 00:58:00,214 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:58:00,214 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,215 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,215 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:58:00,215 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,215 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:58:00,215 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:58:00,215 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:58:00,215 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,215 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:58:00,216 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:00,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:00,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:00,216 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:58:00,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:00,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:00,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:00,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:00,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:00,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:58:00,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:58:00,217 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:58:00,217 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:58:00,218 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:58:00,218 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:58:00,218 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:58:00,218 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,218 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,218 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:00,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:00,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:00,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:00,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:00,219 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:00,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:00,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:00,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:00,220 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:00,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:00,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,221 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:00,221 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:00,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:00,221 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:00,221 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:00,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:58:00,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:58:00,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:58:00,221 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:00,222 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:00,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:00,222 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:58:00,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:00,222 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,222 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:58:00,222 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,222 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,222 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,322 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:58:00,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:58:00,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:58:00,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:58:00,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:58:00,334 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:58:00,334 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:00,334 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:58:00,334 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:58:00,334 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,334 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:00,335 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:00,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:00,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:00,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:00,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,336 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:58:00,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:00,336 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:58:00,337 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,337 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:58:00,337 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:58:00,337 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:58:00,337 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:58:00,337 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:00,337 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:58:00,354 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:58:00,354 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:00,354 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:58:00,354 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:58:00,354 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:58:00,354 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:00,354 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:58:00,354 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:58:00,354 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,354 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:58:00,354 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:58:00,356 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,356 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:00,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:00,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:00,356 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,356 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:58:00,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:00,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:00,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:00,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:58:00,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3371cbf9[Count = 0] remaining members to acquire global barrier 2023-03-20 00:58:00,359 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:58:00,359 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,360 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,360 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:58:00,360 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:58:00,360 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:58:00,361 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,361 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:58:00,362 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:58:00,362 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:00,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:00,362 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:00,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:00,362 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:58:00,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:00,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:00,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:00,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:00,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:58:00,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:58:00,364 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:58:00,364 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:58:00,364 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:58:00,364 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:58:00,365 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:58:00,366 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,366 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:00,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:00,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:00,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,367 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:00,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:00,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:00,368 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:00,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:00,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:00,369 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:00,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:00,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,371 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:00,371 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:00,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:58:00,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:58:00,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:58:00,371 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:00,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:00,371 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:00,372 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:00,372 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,372 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,372 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:58:00,371 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:00,372 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:58:00,372 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,372 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:00,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:00,377 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:00,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49504 deadline: 1679273940375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,472 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:58:00,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:58:00,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:58:00,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:58:00,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:58:00,479 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:58:00,479 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:00,479 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:58:00,479 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:58:00,479 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,479 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:00,480 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,480 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:00,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:00,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:00,480 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,480 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:58:00,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:00,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:58:00,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:00,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:58:00,481 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:58:00,481 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:58:00,481 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:58:00,481 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:00,481 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:58:00,482 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:58:00,482 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:58:00,482 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:00,482 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:58:00,482 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:58:00,482 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:00,482 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:58:00,482 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:58:00,482 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,482 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:58:00,482 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:58:00,484 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:00,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:00,484 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:00,485 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,485 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:58:00,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:00,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:00,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:00,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:58:00,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36bd0b28[Count = 0] remaining members to acquire global barrier 2023-03-20 00:58:00,486 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:58:00,486 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,486 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,486 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:58:00,486 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,486 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:58:00,486 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:58:00,487 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:58:00,487 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,487 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:58:00,487 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:00,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:00,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:00,487 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:58:00,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:00,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:00,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:00,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:00,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:00,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:58:00,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:58:00,489 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:58:00,489 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:58:00,489 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:58:00,489 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:58:00,489 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:58:00,490 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,490 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:00,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:00,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,490 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:00,491 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:00,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:00,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:00,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:00,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:00,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:00,492 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:00,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:00,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,493 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,493 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:00,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:58:00,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:58:00,493 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:00,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:58:00,493 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:00,493 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:00,494 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:58:00,494 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:00,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:00,494 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:00,494 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,494 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,494 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,494 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:58:00,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:00,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:00,583 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/131832804e174057880c4a023d4499fa 2023-03-20 00:58:00,587 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/6fb41291e899495ba3e9528292d314d6 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/6fb41291e899495ba3e9528292d314d6 2023-03-20 00:58:00,590 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/6fb41291e899495ba3e9528292d314d6, entries=200, sequenceid=237, filesize=32.1 K 2023-03-20 00:58:00,590 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-03-20 00:58:00,594 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:58:00,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:58:00,596 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/21038d6bd917471a97dbec7f894f70e9 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/21038d6bd917471a97dbec7f894f70e9 2023-03-20 00:58:00,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:58:00,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:58:00,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:58:00,598 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:58:00,598 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:00,598 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:58:00,598 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:58:00,598 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,598 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:00,599 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:00,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:00,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:00,599 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:00,600 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,600 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:58:00,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:58:00,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:00,600 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:58:00,600 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/21038d6bd917471a97dbec7f894f70e9, entries=150, sequenceid=237, filesize=11.8 K 2023-03-20 00:58:00,601 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:58:00,601 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:58:00,601 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:58:00,601 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:00,601 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:58:00,601 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:58:00,601 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:58:00,601 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/131832804e174057880c4a023d4499fa as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/131832804e174057880c4a023d4499fa 2023-03-20 00:58:00,601 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:00,601 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:58:00,601 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:58:00,601 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:00,601 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:58:00,601 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:58:00,601 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,601 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:58:00,602 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:58:00,602 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,602 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:00,603 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,603 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:58:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:00,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:58:00,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c2ece9a[Count = 0] remaining members to acquire global barrier 2023-03-20 00:58:00,604 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:58:00,604 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,605 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,605 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:58:00,605 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,605 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:58:00,605 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:58:00,605 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:58:00,606 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,606 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:58:00,606 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:00,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:00,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:00,606 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:58:00,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:00,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:00,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:00,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:00,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:00,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:58:00,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:58:00,609 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:58:00,609 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:58:00,609 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:58:00,609 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:58:00,609 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:58:00,610 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,610 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,610 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:00,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:00,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:00,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:00,610 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:00,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:00,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:00,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:00,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:00,612 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:00,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:00,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,613 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:00,613 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:00,613 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:58:00,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:58:00,613 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:00,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:58:00,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:00,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:00,614 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:58:00,614 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:58:00,613 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:00,614 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:00,614 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,614 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,614 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:00,614 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:00,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:00,619 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/131832804e174057880c4a023d4499fa, entries=150, sequenceid=237, filesize=11.8 K 2023-03-20 00:58:00,620 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 3796a7d329c46cabe9e95c4c1fb73632 in 909ms, sequenceid=237, compaction requested=true 2023-03-20 00:58:00,620 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:58:00,620 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:58:00,620 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-20 00:58:00,620 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-20 00:58:00,620 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:58:00,620 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:58:00,626 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60556 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-20 00:58:00,627 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 136846 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-20 00:58:00,627 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 3796a7d329c46cabe9e95c4c1fb73632/B is initiating minor compaction (all files) 2023-03-20 00:58:00,627 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 3796a7d329c46cabe9e95c4c1fb73632/A is initiating minor compaction (all files) 2023-03-20 00:58:00,627 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3796a7d329c46cabe9e95c4c1fb73632/B in TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:00,627 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3796a7d329c46cabe9e95c4c1fb73632/A in TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:00,627 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/aa1ce43eb5fb432f92c3c4ebdc6c96b0, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/5f5cef1394344482b8b67bc172054b55, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/85d763ed2a5442389a68ece12bd82042, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/588cdb929d494cb783e589e32f07a91f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/21038d6bd917471a97dbec7f894f70e9] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp, totalSize=59.1 K 2023-03-20 00:58:00,627 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/1f9ee2f684ea4912809cdffde6c8a42f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/182bb2d36651472cb12b73a0fc2409ed, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/9cc7db9907e14fa3bccbe944df38e00a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/beffb6edf5934c6bb6d05877f9606e6a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/6fb41291e899495ba3e9528292d314d6] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp, totalSize=133.6 K 2023-03-20 00:58:00,627 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting aa1ce43eb5fb432f92c3c4ebdc6c96b0, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1679273875373 2023-03-20 00:58:00,627 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 1f9ee2f684ea4912809cdffde6c8a42f, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1679273875373 2023-03-20 00:58:00,627 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 5f5cef1394344482b8b67bc172054b55, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1679273875499 2023-03-20 00:58:00,627 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 182bb2d36651472cb12b73a0fc2409ed, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1679273875499 2023-03-20 00:58:00,628 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 85d763ed2a5442389a68ece12bd82042, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1679273876629 2023-03-20 00:58:00,628 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 9cc7db9907e14fa3bccbe944df38e00a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1679273876629 2023-03-20 00:58:00,628 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 588cdb929d494cb783e589e32f07a91f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1679273877531 2023-03-20 00:58:00,628 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting beffb6edf5934c6bb6d05877f9606e6a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1679273877531 2023-03-20 00:58:00,628 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 6fb41291e899495ba3e9528292d314d6, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1679273879690 2023-03-20 00:58:00,628 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 21038d6bd917471a97dbec7f894f70e9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1679273879698 2023-03-20 00:58:00,643 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 3796a7d329c46cabe9e95c4c1fb73632#B#compaction#478 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:58:00,659 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3796a7d329c46cabe9e95c4c1fb73632#A#compaction#477 average throughput is 0.99 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:58:00,714 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:58:00,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:58:00,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:58:00,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:58:00,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:58:00,719 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:58:00,719 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:00,719 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:58:00,719 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:58:00,720 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,720 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:00,720 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:00,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:00,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:00,720 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:00,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:00,721 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,721 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:58:00,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:58:00,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:00,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:58:00,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:58:00,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:58:00,722 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:58:00,722 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:00,722 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:58:00,724 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:58:00,724 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:58:00,724 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:00,725 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:58:00,725 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3796a7d329c46cabe9e95c4c1fb73632 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-20 00:58:00,725 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=A 2023-03-20 00:58:00,725 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:58:00,725 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=B 2023-03-20 00:58:00,725 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:58:00,725 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=C 2023-03-20 00:58:00,725 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:58:00,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:58:00,758 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:58:00,780 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:00,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273940777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:00,781 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:00,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273940777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273940778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,782 DEBUG [Thread-3199] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632., hostname=jenkins-hbase20.apache.org,34553,1679273725199, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 00:58:00,783 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:00,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273940780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:00,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49504 deadline: 1679273940880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:00,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273940881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:00,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273940881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:00,887 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:00,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273940884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:01,052 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/10f125b3bfac4ab7a62bf399d68160e7 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/10f125b3bfac4ab7a62bf399d68160e7 2023-03-20 00:58:01,056 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3796a7d329c46cabe9e95c4c1fb73632/B of 3796a7d329c46cabe9e95c4c1fb73632 into 10f125b3bfac4ab7a62bf399d68160e7(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:58:01,056 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:58:01,056 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632., storeName=3796a7d329c46cabe9e95c4c1fb73632/B, priority=11, startTime=1679273880620; duration=0sec 2023-03-20 00:58:01,056 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:58:01,056 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-20 00:58:01,057 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60556 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-20 00:58:01,057 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 3796a7d329c46cabe9e95c4c1fb73632/C is initiating minor compaction (all files) 2023-03-20 00:58:01,057 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3796a7d329c46cabe9e95c4c1fb73632/C in TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:01,057 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/c299cf5972cd4d0fbc4f101e64f882c9, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/a49053a121a64085a103be1cbfc02256, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/cd716c731fbe41db8e52aaa749112c0c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/50c4354bdf024447864015a72e2007b6, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/131832804e174057880c4a023d4499fa] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp, totalSize=59.1 K 2023-03-20 00:58:01,058 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting c299cf5972cd4d0fbc4f101e64f882c9, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1679273875373 2023-03-20 00:58:01,058 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting a49053a121a64085a103be1cbfc02256, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1679273875499 2023-03-20 00:58:01,058 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting cd716c731fbe41db8e52aaa749112c0c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1679273876629 2023-03-20 00:58:01,058 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 50c4354bdf024447864015a72e2007b6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1679273877531 2023-03-20 00:58:01,062 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 131832804e174057880c4a023d4499fa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1679273879698 2023-03-20 00:58:01,068 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 3796a7d329c46cabe9e95c4c1fb73632#C#compaction#480 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:58:01,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:01,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273941085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:01,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:01,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273941086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:01,091 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:01,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273941089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:01,138 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/7c55abecca264e8baea6b216a06d4a0d as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/7c55abecca264e8baea6b216a06d4a0d 2023-03-20 00:58:01,141 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3796a7d329c46cabe9e95c4c1fb73632/A of 3796a7d329c46cabe9e95c4c1fb73632 into 7c55abecca264e8baea6b216a06d4a0d(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:58:01,141 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:58:01,141 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632., storeName=3796a7d329c46cabe9e95c4c1fb73632/A, priority=11, startTime=1679273880620; duration=0sec 2023-03-20 00:58:01,141 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:58:01,147 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303205af86cb4155346b98e6270fc595b4440 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303205af86cb4155346b98e6270fc595b4440 2023-03-20 00:58:01,389 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:01,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273941388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:01,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:01,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273941390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:01,394 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:01,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273941394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:01,476 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/393764122b3a411592c9abe3c2ef6874 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/393764122b3a411592c9abe3c2ef6874 2023-03-20 00:58:01,479 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3796a7d329c46cabe9e95c4c1fb73632/C of 3796a7d329c46cabe9e95c4c1fb73632 into 393764122b3a411592c9abe3c2ef6874(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:58:01,479 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:58:01,479 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632., storeName=3796a7d329c46cabe9e95c4c1fb73632/C, priority=11, startTime=1679273880620; duration=0sec 2023-03-20 00:58:01,479 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:58:01,552 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=251, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/8774e4bf7ab344b5a31a9f9a26264cf3 2023-03-20 00:58:01,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:01,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49504 deadline: 1679273941888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:01,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:01,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273941892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:01,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:01,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273941893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:01,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:01,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273941898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:01,959 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/2159ddd05afe45eeb4dd9a5dc0e9e9ff 2023-03-20 00:58:01,978 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/07b1a309cab44ba0a143e422058a01d9 2023-03-20 00:58:01,982 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/8774e4bf7ab344b5a31a9f9a26264cf3 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/8774e4bf7ab344b5a31a9f9a26264cf3 2023-03-20 00:58:01,984 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/8774e4bf7ab344b5a31a9f9a26264cf3, entries=150, sequenceid=251, filesize=25.3 K 2023-03-20 00:58:01,985 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/2159ddd05afe45eeb4dd9a5dc0e9e9ff as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/2159ddd05afe45eeb4dd9a5dc0e9e9ff 2023-03-20 00:58:01,988 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/2159ddd05afe45eeb4dd9a5dc0e9e9ff, entries=150, sequenceid=251, filesize=11.8 K 2023-03-20 00:58:01,988 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/07b1a309cab44ba0a143e422058a01d9 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/07b1a309cab44ba0a143e422058a01d9 2023-03-20 00:58:01,992 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/07b1a309cab44ba0a143e422058a01d9, entries=150, sequenceid=251, filesize=11.8 K 2023-03-20 00:58:01,992 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 3796a7d329c46cabe9e95c4c1fb73632 in 1267ms, sequenceid=251, compaction requested=false 2023-03-20 00:58:01,992 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:58:01,992 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:01,992 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:58:01,992 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:58:01,992 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:01,992 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:58:01,993 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:58:01,993 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:01,993 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:01,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:01,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:01,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:01,995 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:01,995 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:58:01,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:01,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:01,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:01,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:01,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:01,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:58:01,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b03091[Count = 0] remaining members to acquire global barrier 2023-03-20 00:58:01,997 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:58:01,997 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:01,997 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:01,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:01,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:01,997 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:58:01,998 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:58:01,998 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:58:01,998 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:01,998 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:58:01,998 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:01,998 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:58:01,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:01,998 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:01,999 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:58:02,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:02,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:02,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:02,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:02,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:02,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:02,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:02,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:58:02,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:58:02,016 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:58:02,016 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:58:02,016 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:58:02,016 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:58:02,016 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:58:02,018 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:02,018 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:02,018 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:02,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:02,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:02,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:02,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:02,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:02,019 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:02,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:02,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:02,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:02,020 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:02,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:02,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:02,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:02,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:02,024 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:02,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:58:02,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:02,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:58:02,024 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:02,024 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:02,024 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:02,024 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:02,024 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:02,024 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:02,024 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:02,024 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:58:02,024 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:58:02,025 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:02,025 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:02,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:02,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:02,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:02,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:02,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:02,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:58:02,124 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:58:02,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:58:02,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:58:02,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:58:02,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:58:02,130 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:58:02,130 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:02,130 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:58:02,130 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:58:02,130 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:02,130 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:02,131 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:02,131 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:02,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:02,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:02,131 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:02,132 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:58:02,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:02,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:02,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:58:02,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:02,133 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:58:02,142 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:58:02,142 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:58:02,142 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:58:02,142 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:02,142 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:58:02,150 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:58:02,150 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:02,150 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:58:02,150 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:58:02,151 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3796a7d329c46cabe9e95c4c1fb73632 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-20 00:58:02,151 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=A 2023-03-20 00:58:02,151 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:58:02,151 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=B 2023-03-20 00:58:02,152 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:58:02,152 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=C 2023-03-20 00:58:02,152 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:58:02,577 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320556fcd6af73a4b24a0d3b7e84d6f1fc7 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320556fcd6af73a4b24a0d3b7e84d6f1fc7 2023-03-20 00:58:02,580 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=272, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/a7b0d0b1319341b4ab910231ad6f710e 2023-03-20 00:58:02,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:58:02,897 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:58:02,917 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:02,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273942912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:02,922 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:02,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273942918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:02,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:02,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273942919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:02,987 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/0ffb5b804cd749b2aa0bc96fde9bdb6b 2023-03-20 00:58:02,994 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/1fc0ede45fcd444ca9c1adc2c3ec9b7c 2023-03-20 00:58:02,996 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/a7b0d0b1319341b4ab910231ad6f710e as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/a7b0d0b1319341b4ab910231ad6f710e 2023-03-20 00:58:02,999 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/a7b0d0b1319341b4ab910231ad6f710e, entries=150, sequenceid=272, filesize=25.4 K 2023-03-20 00:58:02,999 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/0ffb5b804cd749b2aa0bc96fde9bdb6b as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/0ffb5b804cd749b2aa0bc96fde9bdb6b 2023-03-20 00:58:03,002 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/0ffb5b804cd749b2aa0bc96fde9bdb6b, entries=150, sequenceid=272, filesize=11.9 K 2023-03-20 00:58:03,003 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/1fc0ede45fcd444ca9c1adc2c3ec9b7c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/1fc0ede45fcd444ca9c1adc2c3ec9b7c 2023-03-20 00:58:03,006 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/1fc0ede45fcd444ca9c1adc2c3ec9b7c, entries=150, sequenceid=272, filesize=11.9 K 2023-03-20 00:58:03,006 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 3796a7d329c46cabe9e95c4c1fb73632 in 855ms, sequenceid=272, compaction requested=true 2023-03-20 00:58:03,007 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:58:03,007 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:03,007 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:58:03,007 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:58:03,007 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,007 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:58:03,007 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:58:03,008 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,008 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:03,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:03,008 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,008 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:58:03,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:03,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:03,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:03,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:03,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:58:03,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fa30d3d[Count = 0] remaining members to acquire global barrier 2023-03-20 00:58:03,009 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:58:03,009 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,010 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,010 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:58:03,010 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:58:03,010 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:58:03,010 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,010 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:58:03,013 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,013 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:58:03,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:03,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:03,013 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:03,013 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:58:03,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:03,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:03,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:03,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:03,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:03,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:58:03,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:58:03,016 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:58:03,016 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:58:03,016 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:58:03,016 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:58:03,016 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:58:03,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:58:03,020 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3796a7d329c46cabe9e95c4c1fb73632 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-20 00:58:03,020 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=A 2023-03-20 00:58:03,020 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:58:03,020 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=B 2023-03-20 00:58:03,020 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:58:03,020 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=C 2023-03-20 00:58:03,020 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:58:03,022 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,022 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,022 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:03,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:03,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:03,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:03,023 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:03,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:03,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:03,025 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:03,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:03,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:03,030 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032060a90b2aaaf04c348baf1fc954dd5e25 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032060a90b2aaaf04c348baf1fc954dd5e25 2023-03-20 00:58:03,031 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,031 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:03,031 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:03,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:58:03,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:58:03,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:03,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:03,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:58:03,032 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:58:03,031 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:03,031 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:03,032 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:58:03,032 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:03,032 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,032 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,032 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,032 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:03,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:03,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:03,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273943039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:03,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273943040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:03,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273943040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,053 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=288, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/1b55b314dc8348e4bc8a2fac726c29d4 2023-03-20 00:58:03,067 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/37d9f1d52796408a9d168ee30ba2c641 2023-03-20 00:58:03,090 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/1a595520a8f64dac8d19c35b4024ea46 2023-03-20 00:58:03,094 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/1b55b314dc8348e4bc8a2fac726c29d4 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/1b55b314dc8348e4bc8a2fac726c29d4 2023-03-20 00:58:03,096 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/1b55b314dc8348e4bc8a2fac726c29d4, entries=150, sequenceid=288, filesize=25.4 K 2023-03-20 00:58:03,097 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/37d9f1d52796408a9d168ee30ba2c641 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/37d9f1d52796408a9d168ee30ba2c641 2023-03-20 00:58:03,099 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/37d9f1d52796408a9d168ee30ba2c641, entries=150, sequenceid=288, filesize=11.9 K 2023-03-20 00:58:03,100 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/1a595520a8f64dac8d19c35b4024ea46 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/1a595520a8f64dac8d19c35b4024ea46 2023-03-20 00:58:03,102 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/1a595520a8f64dac8d19c35b4024ea46, entries=150, sequenceid=288, filesize=11.9 K 2023-03-20 00:58:03,103 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 3796a7d329c46cabe9e95c4c1fb73632 in 83ms, sequenceid=288, compaction requested=true 2023-03-20 00:58:03,103 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:58:03,103 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:58:03,103 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 00:58:03,103 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 00:58:03,103 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:58:03,103 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:58:03,104 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104430 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 00:58:03,104 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 3796a7d329c46cabe9e95c4c1fb73632/A is initiating minor compaction (all files) 2023-03-20 00:58:03,104 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3796a7d329c46cabe9e95c4c1fb73632/A in TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:03,104 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/7c55abecca264e8baea6b216a06d4a0d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/8774e4bf7ab344b5a31a9f9a26264cf3, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/a7b0d0b1319341b4ab910231ad6f710e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/1b55b314dc8348e4bc8a2fac726c29d4] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp, totalSize=102.0 K 2023-03-20 00:58:03,104 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48990 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 00:58:03,105 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 3796a7d329c46cabe9e95c4c1fb73632/B is initiating minor compaction (all files) 2023-03-20 00:58:03,105 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3796a7d329c46cabe9e95c4c1fb73632/B in TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:03,105 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 7c55abecca264e8baea6b216a06d4a0d, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1679273879698 2023-03-20 00:58:03,105 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/10f125b3bfac4ab7a62bf399d68160e7, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/2159ddd05afe45eeb4dd9a5dc0e9e9ff, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/0ffb5b804cd749b2aa0bc96fde9bdb6b, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/37d9f1d52796408a9d168ee30ba2c641] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp, totalSize=47.8 K 2023-03-20 00:58:03,105 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 8774e4bf7ab344b5a31a9f9a26264cf3, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1679273879752 2023-03-20 00:58:03,105 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting a7b0d0b1319341b4ab910231ad6f710e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1679273880772 2023-03-20 00:58:03,105 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 10f125b3bfac4ab7a62bf399d68160e7, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1679273879698 2023-03-20 00:58:03,106 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 1b55b314dc8348e4bc8a2fac726c29d4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1679273882916 2023-03-20 00:58:03,106 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 2159ddd05afe45eeb4dd9a5dc0e9e9ff, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1679273879752 2023-03-20 00:58:03,107 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 0ffb5b804cd749b2aa0bc96fde9bdb6b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1679273880772 2023-03-20 00:58:03,108 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 37d9f1d52796408a9d168ee30ba2c641, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1679273882916 2023-03-20 00:58:03,124 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 3796a7d329c46cabe9e95c4c1fb73632#B#compaction#490 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:58:03,131 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3796a7d329c46cabe9e95c4c1fb73632#A#compaction#489 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:58:03,132 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:58:03,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:58:03,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:58:03,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:58:03,146 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3796a7d329c46cabe9e95c4c1fb73632 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-20 00:58:03,146 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=A 2023-03-20 00:58:03,146 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:58:03,146 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=B 2023-03-20 00:58:03,146 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:58:03,146 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=C 2023-03-20 00:58:03,146 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:58:03,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:58:03,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:58:03,150 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:58:03,150 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:03,150 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:58:03,150 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:58:03,150 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,151 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:03,151 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:03,151 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:03,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:03,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:03,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,152 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,152 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:58:03,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:58:03,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:03,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:58:03,162 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:58:03,162 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:58:03,171 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:58:03,171 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:03,171 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:58:03,171 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:58:03,171 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:58:03,172 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:03,172 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:58:03,172 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:58:03,172 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:03,172 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:58:03,172 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:58:03,172 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,172 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:58:03,172 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:58:03,173 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:03,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:03,173 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,174 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,174 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:58:03,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:03,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:03,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:03,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:03,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:58:03,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6199b5aa[Count = 0] remaining members to acquire global barrier 2023-03-20 00:58:03,175 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:58:03,175 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,175 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,175 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:58:03,175 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:58:03,175 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:58:03,175 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,175 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:58:03,177 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:03,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:03,177 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:58:03,177 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:03,177 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:58:03,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:03,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:03,177 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:03,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273943173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:03,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:03,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:03,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:58:03,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:58:03,179 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:58:03,179 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:58:03,179 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:58:03,179 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:58:03,179 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:58:03,179 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,179 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:03,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,180 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:03,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:03,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:03,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:03,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:03,180 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:03,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:03,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:03,181 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:03,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273943177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:03,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273943177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:03,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:03,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,183 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,183 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:03,183 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:03,183 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,184 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,184 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:03,184 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:03,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:03,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:58:03,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:58:03,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:58:03,184 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,184 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:03,184 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:03,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:03,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:03,184 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:58:03,184 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:58:03,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:03,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273943278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,284 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:03,284 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:58:03,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49438 deadline: 1679273943283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,284 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:03,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49480 deadline: 1679273943283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:58:03,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:58:03,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:58:03,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:58:03,289 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:58:03,289 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:03,289 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:58:03,289 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:58:03,289 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,289 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:03,290 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:03,290 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:03,290 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,290 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:58:03,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:58:03,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:03,291 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:58:03,291 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:58:03,292 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:58:03,292 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:58:03,292 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:03,292 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:58:03,292 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:58:03,292 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:58:03,292 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:03,292 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:58:03,292 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:58:03,292 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:03,292 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:58:03,292 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:58:03,292 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,292 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:58:03,292 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:58:03,293 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,293 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:03,293 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,293 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:58:03,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:03,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:03,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:03,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:58:03,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b39fd1d[Count = 0] remaining members to acquire global barrier 2023-03-20 00:58:03,296 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:58:03,296 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,296 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,297 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,297 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:58:03,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,297 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:58:03,297 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:58:03,297 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:58:03,298 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,298 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:58:03,298 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:03,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:03,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:03,298 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:58:03,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:03,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:03,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:03,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:03,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:03,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:58:03,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:58:03,302 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:58:03,302 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:58:03,302 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:58:03,302 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:58:03,302 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:58:03,308 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,308 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:03,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,308 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:03,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:03,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:03,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:03,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:03,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:03,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:03,310 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:03,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:03,311 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:03,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,316 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:03,316 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:03,316 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:03,316 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:03,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:58:03,316 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:03,316 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:03,316 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,316 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,317 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,317 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:58:03,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:58:03,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:03,317 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:58:03,317 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:58:03,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:03,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:03,417 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:58:03,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:58:03,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:58:03,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:58:03,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:58:03,420 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:58:03,420 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:03,420 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:58:03,420 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:58:03,420 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,420 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:03,421 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,421 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:03,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:03,421 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,421 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:58:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:03,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:58:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:03,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:58:03,424 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:58:03,424 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:58:03,424 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:58:03,424 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:03,424 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:58:03,424 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:58:03,424 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:58:03,424 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:03,424 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:58:03,424 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:58:03,424 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:03,425 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:58:03,425 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:58:03,425 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,425 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:58:03,425 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:58:03,426 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,426 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:03,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:03,426 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,426 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:58:03,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:03,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:03,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:03,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:03,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:58:03,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@180e6253[Count = 0] remaining members to acquire global barrier 2023-03-20 00:58:03,429 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:58:03,429 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,429 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,429 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:58:03,429 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:58:03,429 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:58:03,430 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,430 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:58:03,431 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:58:03,431 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:03,431 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,431 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:58:03,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:03,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:03,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:03,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:03,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:03,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:03,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:03,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:58:03,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:58:03,434 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:58:03,434 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:58:03,434 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:58:03,434 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:58:03,434 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:58:03,436 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,436 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:03,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:03,436 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:03,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:03,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:03,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:03,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,438 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:03,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:03,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:03,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,439 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:03,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:03,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:03,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:03,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:03,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,441 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:03,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:03,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:58:03,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:58:03,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:58:03,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:03,441 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:58:03,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:03,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,441 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,441 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:58:03,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:03,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:03,483 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:03,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49468 deadline: 1679273943482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:03,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49480 deadline: 1679273943486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:03,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49438 deadline: 1679273943486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,541 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/9561670ffc9f4d809f8e5f968e699a69 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/9561670ffc9f4d809f8e5f968e699a69 2023-03-20 00:58:03,542 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:58:03,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:58:03,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:58:03,552 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3796a7d329c46cabe9e95c4c1fb73632/B of 3796a7d329c46cabe9e95c4c1fb73632 into 9561670ffc9f4d809f8e5f968e699a69(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:58:03,552 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:58:03,552 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632., storeName=3796a7d329c46cabe9e95c4c1fb73632/B, priority=12, startTime=1679273883103; duration=0sec 2023-03-20 00:58:03,552 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:58:03,552 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 00:58:03,553 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48990 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 00:58:03,553 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 3796a7d329c46cabe9e95c4c1fb73632/C is initiating minor compaction (all files) 2023-03-20 00:58:03,553 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3796a7d329c46cabe9e95c4c1fb73632/C in TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:03,553 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/393764122b3a411592c9abe3c2ef6874, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/07b1a309cab44ba0a143e422058a01d9, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/1fc0ede45fcd444ca9c1adc2c3ec9b7c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/1a595520a8f64dac8d19c35b4024ea46] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp, totalSize=47.8 K 2023-03-20 00:58:03,554 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 393764122b3a411592c9abe3c2ef6874, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1679273879698 2023-03-20 00:58:03,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:58:03,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:58:03,554 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:58:03,554 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:03,554 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:58:03,554 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:58:03,554 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 07b1a309cab44ba0a143e422058a01d9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1679273879752 2023-03-20 00:58:03,554 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,554 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:03,555 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 1fc0ede45fcd444ca9c1adc2c3ec9b7c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1679273880772 2023-03-20 00:58:03,555 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:03,555 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:03,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:03,555 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 1a595520a8f64dac8d19c35b4024ea46, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1679273882916 2023-03-20 00:58:03,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:03,557 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,557 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:58:03,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:58:03,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:03,557 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:58:03,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:58:03,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:58:03,578 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:58:03,578 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:03,578 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:58:03,588 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 3796a7d329c46cabe9e95c4c1fb73632#C#compaction#492 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:58:03,590 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:58:03,590 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:58:03,594 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:03,594 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:58:03,595 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:58:03,595 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:03,595 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:58:03,595 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:58:03,595 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,602 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:58:03,602 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:58:03,603 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,603 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:03,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:03,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:03,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:03,604 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,604 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:58:03,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:03,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:03,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:58:03,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f635e8a[Count = 0] remaining members to acquire global barrier 2023-03-20 00:58:03,605 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:58:03,606 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,606 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,606 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:58:03,606 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:58:03,606 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:58:03,606 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,607 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:58:03,608 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,608 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:58:03,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:03,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:03,608 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:03,608 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:58:03,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:03,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:03,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:03,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:03,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:03,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:58:03,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:58:03,611 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:58:03,611 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:58:03,612 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:58:03,612 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:58:03,612 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:58:03,613 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,613 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:03,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,613 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:03,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:03,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:03,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:03,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:03,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:03,614 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:03,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:03,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:03,616 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:03,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,618 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:03,618 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:03,618 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:58:03,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:58:03,618 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:58:03,618 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:58:03,618 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:03,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:03,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:03,618 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:03,618 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:03,618 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,618 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,618 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:03,618 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:03,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:03,622 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320e1ffcdbffc154535937ed329d64dce30 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320e1ffcdbffc154535937ed329d64dce30 2023-03-20 00:58:03,642 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=306, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/2a080519b7604d4eb4e7203f64ac319e 2023-03-20 00:58:03,661 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/052545d1c65f40098b5a1791a40e7abd 2023-03-20 00:58:03,677 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/a8cd672775464b3da4f742b53023c90c 2023-03-20 00:58:03,681 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/2a080519b7604d4eb4e7203f64ac319e as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/2a080519b7604d4eb4e7203f64ac319e 2023-03-20 00:58:03,684 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/2a080519b7604d4eb4e7203f64ac319e, entries=250, sequenceid=306, filesize=39.2 K 2023-03-20 00:58:03,684 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/052545d1c65f40098b5a1791a40e7abd as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/052545d1c65f40098b5a1791a40e7abd 2023-03-20 00:58:03,687 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/052545d1c65f40098b5a1791a40e7abd, entries=150, sequenceid=306, filesize=11.9 K 2023-03-20 00:58:03,688 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/a8cd672775464b3da4f742b53023c90c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/a8cd672775464b3da4f742b53023c90c 2023-03-20 00:58:03,691 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/a8cd672775464b3da4f742b53023c90c, entries=150, sequenceid=306, filesize=11.9 K 2023-03-20 00:58:03,691 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 3796a7d329c46cabe9e95c4c1fb73632 in 545ms, sequenceid=306, compaction requested=false 2023-03-20 00:58:03,691 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:58:03,718 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:58:03,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:58:03,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:58:03,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:58:03,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:58:03,721 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:58:03,721 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:03,721 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:58:03,721 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:58:03,721 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,721 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:03,722 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:03,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:03,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:03,722 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:03,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:03,723 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,723 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:58:03,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:58:03,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:03,723 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:58:03,726 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:58:03,726 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:58:03,726 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:58:03,726 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:03,726 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:58:03,726 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:58:03,726 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:58:03,726 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:03,727 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:58:03,727 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3796a7d329c46cabe9e95c4c1fb73632 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-20 00:58:03,727 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=A 2023-03-20 00:58:03,727 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:58:03,727 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=B 2023-03-20 00:58:03,727 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:58:03,727 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=C 2023-03-20 00:58:03,727 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:58:03,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:58:03,789 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:58:03,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:03,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49480 deadline: 1679273943806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,809 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:03,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49468 deadline: 1679273943806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,809 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:03,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49438 deadline: 1679273943807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:03,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49504 deadline: 1679273943906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,908 DEBUG [Thread-3197] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632., hostname=jenkins-hbase20.apache.org,34553,1679273725199, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 00:58:03,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:03,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49480 deadline: 1679273943909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:03,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49468 deadline: 1679273943909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:03,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:03,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49438 deadline: 1679273943910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:04,033 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/8c59efd8d024430aa9fb6853e76e65bd as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/8c59efd8d024430aa9fb6853e76e65bd 2023-03-20 00:58:04,036 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3796a7d329c46cabe9e95c4c1fb73632/C of 3796a7d329c46cabe9e95c4c1fb73632 into 8c59efd8d024430aa9fb6853e76e65bd(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:58:04,037 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:58:04,037 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632., storeName=3796a7d329c46cabe9e95c4c1fb73632/C, priority=12, startTime=1679273883103; duration=0sec 2023-03-20 00:58:04,037 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:58:04,063 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/0492140a091f42f8a467bef618498e1a as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/0492140a091f42f8a467bef618498e1a 2023-03-20 00:58:04,066 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3796a7d329c46cabe9e95c4c1fb73632/A of 3796a7d329c46cabe9e95c4c1fb73632 into 0492140a091f42f8a467bef618498e1a(size=26.1 K), total size for store is 65.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:58:04,066 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:58:04,067 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632., storeName=3796a7d329c46cabe9e95c4c1fb73632/A, priority=12, startTime=1679273883103; duration=0sec 2023-03-20 00:58:04,067 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:58:04,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:04,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49480 deadline: 1679273944111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:04,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:04,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49468 deadline: 1679273944111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:04,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:04,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49438 deadline: 1679273944112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:04,144 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320cd3b493dce6b45c8a345bd8ece5d66dd to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320cd3b493dce6b45c8a345bd8ece5d66dd 2023-03-20 00:58:04,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:04,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49480 deadline: 1679273944415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:04,418 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:04,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49438 deadline: 1679273944415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:04,418 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:04,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49468 deadline: 1679273944416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:04,549 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=322, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/7ce59681697c47ad889430261153ac0b 2023-03-20 00:58:04,578 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/4970105f688d49a1b483d03b14c0c89f 2023-03-20 00:58:04,588 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/19153e7eea48497986a8a7f97c8a8082 2023-03-20 00:58:04,591 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/7ce59681697c47ad889430261153ac0b as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/7ce59681697c47ad889430261153ac0b 2023-03-20 00:58:04,593 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/7ce59681697c47ad889430261153ac0b, entries=150, sequenceid=322, filesize=25.4 K 2023-03-20 00:58:04,594 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/4970105f688d49a1b483d03b14c0c89f as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/4970105f688d49a1b483d03b14c0c89f 2023-03-20 00:58:04,597 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/4970105f688d49a1b483d03b14c0c89f, entries=150, sequenceid=322, filesize=11.9 K 2023-03-20 00:58:04,597 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/19153e7eea48497986a8a7f97c8a8082 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/19153e7eea48497986a8a7f97c8a8082 2023-03-20 00:58:04,601 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/19153e7eea48497986a8a7f97c8a8082, entries=150, sequenceid=322, filesize=11.9 K 2023-03-20 00:58:04,601 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 3796a7d329c46cabe9e95c4c1fb73632 in 874ms, sequenceid=322, compaction requested=true 2023-03-20 00:58:04,602 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:58:04,602 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:04,602 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:58:04,602 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:58:04,602 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:04,602 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:58:04,602 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:58:04,603 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:04,603 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:04,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:04,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:04,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:04,603 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:04,603 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:58:04,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:04,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:04,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:04,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:04,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:04,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:58:04,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13fb8829[Count = 0] remaining members to acquire global barrier 2023-03-20 00:58:04,605 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:58:04,605 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:04,606 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:04,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:04,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:04,606 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:58:04,606 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:58:04,606 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:58:04,606 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:04,606 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:58:04,607 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:58:04,607 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:04,607 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:58:04,607 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:04,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:04,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:04,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:04,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:04,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:04,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:04,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:04,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:04,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:04,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:04,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:58:04,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:58:04,609 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:58:04,609 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:58:04,609 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:58:04,609 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:58:04,609 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:58:04,611 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:04,611 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:04,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:04,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:04,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:04,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:04,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:04,611 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:04,611 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:04,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:04,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:04,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:04,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:04,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:04,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:04,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:04,613 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:04,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:04,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:04,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:04,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:04,615 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:04,615 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:04,615 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:04,615 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:04,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:58:04,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:04,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:04,615 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:04,615 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:04,615 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:04,615 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:04,615 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:04,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:58:04,615 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:04,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:04,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:04,616 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:58:04,616 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:58:04,716 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:58:04,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:58:04,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:58:04,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:58:04,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:58:04,718 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:58:04,718 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:04,718 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:58:04,719 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:58:04,719 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:04,719 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:04,719 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:04,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:04,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:04,719 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:04,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:04,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:04,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:58:04,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:04,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:58:04,720 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:58:04,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:58:04,721 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:58:04,721 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:04,721 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:58:04,721 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:58:04,721 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:58:04,721 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:04,721 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:58:04,721 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3796a7d329c46cabe9e95c4c1fb73632 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-20 00:58:04,721 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=A 2023-03-20 00:58:04,721 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:58:04,721 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=B 2023-03-20 00:58:04,721 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:58:04,721 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=C 2023-03-20 00:58:04,721 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:58:04,722 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:04,722 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:58:04,820 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:58:04,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:58:04,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:04,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273944873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:04,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:04,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49480 deadline: 1679273944921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:04,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:04,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49468 deadline: 1679273944922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:04,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:04,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49438 deadline: 1679273944924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:04,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:04,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273944977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,140 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303201637260ae06e4ee9a6bc0ded3f612226 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303201637260ae06e4ee9a6bc0ded3f612226 2023-03-20 00:58:05,166 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=341, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/a98b27075b00409f800d7fae992887be 2023-03-20 00:58:05,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:05,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273945181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,208 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=341 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/dafabe814e394d1b858104d29a7cd519 2023-03-20 00:58:05,486 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:05,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273945484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,620 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=341 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/ce5963caedd1472d8a4f283ab550fdd0 2023-03-20 00:58:05,623 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/a98b27075b00409f800d7fae992887be as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/a98b27075b00409f800d7fae992887be 2023-03-20 00:58:05,626 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/a98b27075b00409f800d7fae992887be, entries=150, sequenceid=341, filesize=25.4 K 2023-03-20 00:58:05,626 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/dafabe814e394d1b858104d29a7cd519 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/dafabe814e394d1b858104d29a7cd519 2023-03-20 00:58:05,629 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/dafabe814e394d1b858104d29a7cd519, entries=150, sequenceid=341, filesize=11.9 K 2023-03-20 00:58:05,629 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/ce5963caedd1472d8a4f283ab550fdd0 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/ce5963caedd1472d8a4f283ab550fdd0 2023-03-20 00:58:05,632 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/ce5963caedd1472d8a4f283ab550fdd0, entries=150, sequenceid=341, filesize=11.9 K 2023-03-20 00:58:05,633 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 3796a7d329c46cabe9e95c4c1fb73632 in 912ms, sequenceid=341, compaction requested=true 2023-03-20 00:58:05,633 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:58:05,633 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:05,633 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:58:05,633 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:58:05,633 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,633 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:58:05,633 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:58:05,634 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,634 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:05,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:05,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:05,634 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:05,634 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:58:05,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:05,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:05,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:05,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:05,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:58:05,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@477e5e5e[Count = 0] remaining members to acquire global barrier 2023-03-20 00:58:05,636 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:58:05,636 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:05,636 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:05,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:05,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:05,636 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:58:05,636 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:58:05,636 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,636 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:58:05,636 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:58:05,638 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:58:05,638 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,638 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:05,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:05,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:05,638 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:58:05,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:05,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:05,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:05,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:05,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:05,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:58:05,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:58:05,640 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:58:05,640 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:58:05,640 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:58:05,640 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:58:05,640 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:58:05,641 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:05,641 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:05,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:05,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:05,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:05,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:05,641 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:05,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:05,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:05,641 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:05,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:05,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:05,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:05,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:05,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:05,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,642 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:05,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:05,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,646 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,646 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:05,646 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:05,646 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,646 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:05,646 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:05,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:58:05,646 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:05,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:58:05,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:05,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:05,646 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:05,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:58:05,646 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:05,646 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:05,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:05,647 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:58:05,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:05,647 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:58:05,747 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:58:05,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:58:05,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:58:05,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:58:05,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:58:05,755 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:58:05,755 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:05,755 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:58:05,755 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:58:05,755 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:05,755 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:05,756 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:05,756 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:05,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:05,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:05,756 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,756 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:58:05,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:05,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:58:05,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:05,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:58:05,762 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:58:05,762 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:58:05,762 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:58:05,762 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:05,762 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:58:05,768 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:58:05,768 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:58:05,768 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:05,768 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:58:05,768 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3796a7d329c46cabe9e95c4c1fb73632 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-20 00:58:05,769 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=A 2023-03-20 00:58:05,769 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:58:05,769 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=B 2023-03-20 00:58:05,769 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:58:05,769 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=C 2023-03-20 00:58:05,769 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:58:05,781 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320db3fff6821ed4c819c9af33d92e8bf84 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320db3fff6821ed4c819c9af33d92e8bf84 2023-03-20 00:58:05,801 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=356, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/711dc43500d94ff4a77a6c64cd410153 2023-03-20 00:58:05,815 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/ef5d565fca1d4794b8b70762196e3ac1 2023-03-20 00:58:05,844 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/a8364ac210ef48819bf3b4cb9b0fb127 2023-03-20 00:58:05,848 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/711dc43500d94ff4a77a6c64cd410153 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/711dc43500d94ff4a77a6c64cd410153 2023-03-20 00:58:05,852 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/711dc43500d94ff4a77a6c64cd410153, entries=150, sequenceid=356, filesize=25.4 K 2023-03-20 00:58:05,853 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/ef5d565fca1d4794b8b70762196e3ac1 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/ef5d565fca1d4794b8b70762196e3ac1 2023-03-20 00:58:05,856 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/ef5d565fca1d4794b8b70762196e3ac1, entries=150, sequenceid=356, filesize=11.9 K 2023-03-20 00:58:05,857 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/a8364ac210ef48819bf3b4cb9b0fb127 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/a8364ac210ef48819bf3b4cb9b0fb127 2023-03-20 00:58:05,861 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/a8364ac210ef48819bf3b4cb9b0fb127, entries=150, sequenceid=356, filesize=11.9 K 2023-03-20 00:58:05,865 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 3796a7d329c46cabe9e95c4c1fb73632 in 97ms, sequenceid=356, compaction requested=true 2023-03-20 00:58:05,865 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:58:05,865 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:05,865 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:58:05,865 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:58:05,865 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,865 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:58:05,865 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:58:05,867 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:05,867 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:05,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:05,867 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:05,868 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:58:05,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:05,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:05,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:05,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:05,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:58:05,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c89a018[Count = 0] remaining members to acquire global barrier 2023-03-20 00:58:05,870 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:58:05,870 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:05,870 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:05,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:05,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:05,870 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:58:05,870 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,870 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:58:05,870 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:58:05,871 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:58:05,871 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:58:05,871 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:05,871 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,871 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:58:05,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:05,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:05,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:05,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:05,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:05,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:05,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:05,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:58:05,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:58:05,873 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:58:05,873 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:58:05,873 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:58:05,873 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:58:05,874 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:58:05,874 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:05,874 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:05,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:05,875 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:05,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:05,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:05,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:05,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:05,875 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:05,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:05,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:05,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:05,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:05,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:05,876 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:05,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:05,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:05,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,877 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,877 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:05,877 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:05,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:58:05,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:58:05,877 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:05,877 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,877 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:05,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:05,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:05,877 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:05,877 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:05,878 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:05,877 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:58:05,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:58:05,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:05,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:05,878 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:58:05,878 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:05,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:58:05,960 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3796a7d329c46cabe9e95c4c1fb73632 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-20 00:58:05,961 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=A 2023-03-20 00:58:05,961 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:58:05,961 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=B 2023-03-20 00:58:05,961 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:58:05,961 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=C 2023-03-20 00:58:05,961 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:58:05,978 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:58:05,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:58:05,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:58:05,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:58:05,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:58:05,982 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:58:05,982 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:05,982 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:58:05,982 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:58:05,983 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:05,983 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:05,984 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:05,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:05,984 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:05,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:05,984 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,984 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:58:05,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:05,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:58:05,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:05,985 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:58:05,985 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:58:05,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:58:05,985 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:58:05,985 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:05,986 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:58:05,986 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:58:05,986 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:58:05,986 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:05,986 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:58:05,986 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:58:05,986 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:05,986 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:58:05,986 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:58:05,986 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,986 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:58:05,986 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:58:05,987 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,987 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:05,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:05,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:05,988 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:05,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:05,988 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:58:05,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:05,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:05,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:05,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:58:05,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@93b3c0d[Count = 0] remaining members to acquire global barrier 2023-03-20 00:58:05,989 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:58:05,989 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:05,990 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:05,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:05,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:05,990 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:58:05,990 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,991 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:58:05,990 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:58:05,991 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:58:05,991 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:58:05,991 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,991 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:05,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:05,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:05,992 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:58:05,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:05,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:05,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:05,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:05,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:05,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:58:05,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:58:05,994 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:58:05,994 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:58:05,994 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:58:05,994 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:58:05,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:05,994 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:58:05,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49480 deadline: 1679273945990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679273885979, max timestamp=1679273885993Num uniques 859; 2023-03-20 00:58:05,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679273885979, max timestamp=1679273885993Num uniques 859; 2023-03-20 00:58:05,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 00:58:05,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 00:58:05,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679273885979, max timestamp=1679273885993Num uniques 859; 2023-03-20 00:58:05,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 00:58:05,995 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:05,995 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:05,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:05,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:05,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:05,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:05,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:05,995 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:05,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:05,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:05,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:05,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:05,996 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:05,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,997 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:05,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,999 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,999 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:05,999 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:05,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:58:05,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:58:05,999 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:05,999 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:05,999 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:05,999 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:05,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:05,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:05,999 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:05,999 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:58:05,999 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:58:05,999 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:05,999 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:06,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:06,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:06,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49468 deadline: 1679273946011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:06,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273946013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:06,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49438 deadline: 1679273946013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:06,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49480 deadline: 1679273946095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,100 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:58:06,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:58:06,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:58:06,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:58:06,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:58:06,103 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:58:06,103 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:06,103 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:58:06,103 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:58:06,104 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,104 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,104 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:06,104 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:06,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:06,105 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,105 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:58:06,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:58:06,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,105 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:58:06,106 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:58:06,106 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:58:06,106 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:58:06,106 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:06,106 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:58:06,106 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:58:06,106 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:58:06,106 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:06,106 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:58:06,106 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:58:06,106 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:06,106 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:58:06,106 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:58:06,106 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,106 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:58:06,106 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:58:06,107 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,107 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:06,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:06,107 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,107 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:58:06,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:06,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:06,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:06,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:58:06,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46ce47c3[Count = 0] remaining members to acquire global barrier 2023-03-20 00:58:06,108 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:58:06,108 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,109 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,109 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,109 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:58:06,109 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:58:06,109 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:58:06,109 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:58:06,110 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,110 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:58:06,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:06,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:06,110 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:06,110 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:58:06,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:06,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:06,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:06,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:58:06,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:58:06,111 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:58:06,111 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:58:06,111 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:58:06,111 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:58:06,112 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:58:06,112 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:06,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:06,112 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,112 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:06,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:06,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:06,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:06,113 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:06,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,114 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:06,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:06,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49468 deadline: 1679273946114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,116 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,116 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:06,116 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,116 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:06,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:58:06,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:58:06,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:06,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:06,116 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:06,116 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,117 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:58:06,117 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:58:06,117 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,117 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,117 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,117 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:06,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:06,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:06,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273946117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:06,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49438 deadline: 1679273946117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,217 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:58:06,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:58:06,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:58:06,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:58:06,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:58:06,219 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:58:06,219 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:06,220 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:58:06,220 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:58:06,220 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,220 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,220 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,221 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:06,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:06,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:06,221 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,221 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:58:06,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:58:06,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:58:06,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:58:06,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:58:06,222 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:58:06,222 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:06,222 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:58:06,222 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:58:06,222 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:58:06,222 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:06,222 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:58:06,222 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:58:06,222 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:06,222 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:58:06,222 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:58:06,222 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,222 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:58:06,222 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:58:06,223 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,223 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:06,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:06,223 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,223 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:58:06,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:06,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:06,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:06,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:58:06,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73776dea[Count = 0] remaining members to acquire global barrier 2023-03-20 00:58:06,225 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:58:06,225 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,225 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,226 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:58:06,226 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:58:06,226 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:58:06,226 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,226 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:58:06,226 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,226 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:58:06,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:06,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:06,226 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:06,226 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:58:06,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:06,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:06,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:06,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:58:06,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:58:06,228 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:58:06,228 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:58:06,228 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:58:06,228 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:58:06,228 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:58:06,229 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,229 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:06,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:06,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,229 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:06,229 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:06,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:06,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:06,230 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:06,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,231 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,231 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,231 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:06,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:58:06,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:58:06,231 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:06,231 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:06,232 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:06,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:06,231 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:06,232 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:58:06,232 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:58:06,232 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,232 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:06,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:06,232 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:58:06,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:06,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49480 deadline: 1679273946298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:06,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49468 deadline: 1679273946317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:06,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273946320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:06,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49438 deadline: 1679273946320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,332 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:58:06,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:58:06,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:58:06,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:58:06,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:58:06,335 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:58:06,335 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:06,335 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:58:06,335 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:58:06,335 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,335 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,336 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:06,336 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:06,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:06,336 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,336 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:58:06,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:58:06,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:58:06,339 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:58:06,339 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:58:06,339 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:58:06,339 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:06,339 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:58:06,339 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:58:06,339 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:58:06,339 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:06,340 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:58:06,340 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:58:06,340 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:06,340 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:58:06,340 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:58:06,340 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,340 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:58:06,340 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:58:06,341 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,341 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:06,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:06,341 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,341 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:58:06,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:06,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:06,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:06,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:58:06,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@129f8afc[Count = 0] remaining members to acquire global barrier 2023-03-20 00:58:06,342 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:58:06,342 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,343 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,343 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:58:06,343 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:58:06,343 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:58:06,343 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,343 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:58:06,344 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,344 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:58:06,344 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:06,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:06,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:06,344 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:58:06,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:06,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:06,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:06,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:58:06,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:58:06,346 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:58:06,346 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:58:06,346 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:58:06,346 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:58:06,346 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:58:06,346 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,346 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:06,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:06,347 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:06,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:06,347 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:06,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:06,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:06,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,348 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:06,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,349 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,349 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:06,349 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:06,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:06,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:58:06,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:58:06,349 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:06,349 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:06,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:58:06,350 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:58:06,349 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,350 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:58:06,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:06,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:06,350 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,350 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,350 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,350 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,381 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320f223a7b1230c460baae5d5985547b36c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320f223a7b1230c460baae5d5985547b36c 2023-03-20 00:58:06,450 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:58:06,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:58:06,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:58:06,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:58:06,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:58:06,452 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:58:06,452 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:06,452 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:58:06,452 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:58:06,453 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,453 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,453 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:06,453 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:06,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:06,454 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,454 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:58:06,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:58:06,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,454 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:58:06,455 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:58:06,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:58:06,455 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:58:06,455 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:06,455 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:58:06,455 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:58:06,455 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:58:06,455 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:06,455 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:58:06,455 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:58:06,455 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:06,455 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:58:06,455 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:58:06,455 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,455 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:58:06,455 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:58:06,456 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,456 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:06,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:06,456 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,456 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:58:06,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:06,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:06,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:06,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:58:06,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@429407f6[Count = 0] remaining members to acquire global barrier 2023-03-20 00:58:06,457 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:58:06,458 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,458 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,458 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:58:06,458 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:58:06,458 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:58:06,458 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,458 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:58:06,459 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:58:06,459 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:06,459 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,459 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:58:06,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:06,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:06,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:06,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:06,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:06,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:58:06,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:58:06,461 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:58:06,461 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:58:06,461 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:58:06,461 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:58:06,461 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:58:06,462 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,462 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:06,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:06,462 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:06,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:06,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:06,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:06,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,463 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:06,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,464 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:06,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,465 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,465 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,465 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,465 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:06,465 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:06,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:06,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:06,465 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,465 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,465 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:58:06,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:06,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:06,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:58:06,465 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,465 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:06,465 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:58:06,465 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:58:06,566 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:58:06,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:58:06,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:58:06,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:58:06,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:58:06,568 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:58:06,568 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:06,568 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:58:06,568 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:58:06,568 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,569 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,569 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,569 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:06,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:06,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:06,569 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,569 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:58:06,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:58:06,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:58:06,570 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:58:06,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:58:06,570 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:58:06,570 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:06,570 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:58:06,571 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:58:06,571 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:58:06,571 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:06,571 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:58:06,571 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:58:06,571 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:06,571 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:58:06,571 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:58:06,571 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,571 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:58:06,571 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:58:06,572 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,573 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,573 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:58:06,573 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:06,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:06,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:06,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:06,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:06,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:58:06,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7650b038[Count = 0] remaining members to acquire global barrier 2023-03-20 00:58:06,574 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:58:06,574 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,574 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,575 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,575 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:58:06,575 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:58:06,575 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:58:06,575 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:58:06,575 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,575 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:58:06,575 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:06,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:06,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:06,575 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:58:06,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:06,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:06,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:06,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:58:06,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:58:06,577 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:58:06,577 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:58:06,577 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:58:06,577 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:58:06,577 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:58:06,578 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,578 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:06,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:06,578 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:06,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:06,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:06,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:06,578 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:06,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,579 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:06,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,581 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:06,581 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:58:06,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:58:06,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:58:06,581 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,581 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:06,581 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:58:06,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:06,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:06,581 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:06,581 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:58:06,581 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,581 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,581 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,581 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,581 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:06,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:06,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:06,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49480 deadline: 1679273946602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:06,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49438 deadline: 1679273946624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:06,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49468 deadline: 1679273946624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:06,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273946624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,681 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:58:06,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:58:06,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:58:06,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:58:06,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:58:06,686 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:58:06,686 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:06,686 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:58:06,686 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:58:06,686 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,687 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,687 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:06,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:06,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:06,687 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,688 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,688 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:58:06,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:58:06,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,688 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:58:06,689 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:58:06,689 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:58:06,689 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:58:06,689 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:06,689 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:58:06,689 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:58:06,689 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:58:06,689 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:06,689 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:58:06,690 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:58:06,690 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:06,690 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:58:06,690 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:58:06,690 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,690 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:58:06,690 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:58:06,691 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,691 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:06,691 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:06,691 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:58:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:06,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:06,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:58:06,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e808b25[Count = 0] remaining members to acquire global barrier 2023-03-20 00:58:06,693 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:58:06,693 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,693 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,694 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:58:06,694 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:58:06,694 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:58:06,694 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,694 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:58:06,695 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:06,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:06,695 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:58:06,695 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:06,695 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:58:06,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:06,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:06,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:06,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:58:06,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:58:06,697 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:58:06,697 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:58:06,697 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:58:06,697 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:58:06,697 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:58:06,698 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,698 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:06,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,698 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:06,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:06,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:06,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:06,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:06,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,699 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:06,700 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:06,702 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,702 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,702 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,702 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,702 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:06,702 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:06,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:06,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:58:06,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:58:06,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:06,702 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:58:06,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:58:06,702 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:06,702 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:06,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:06,703 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,703 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,703 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:58:06,786 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=367, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/55e51e67b9364e1babb8a9cec40707ae 2023-03-20 00:58:06,803 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:58:06,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:58:06,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:58:06,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:58:06,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:58:06,807 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:58:06,808 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:06,808 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:58:06,808 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:58:06,808 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,808 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,809 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,809 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:06,809 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,809 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:58:06,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:06,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:06,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:58:06,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,810 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:58:06,811 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:58:06,811 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:58:06,811 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:06,811 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:58:06,832 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:58:06,832 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:06,832 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:58:06,832 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:58:06,833 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:58:06,833 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:06,833 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:58:06,833 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:58:06,833 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,833 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:58:06,833 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:58:06,941 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,941 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:06,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:06,946 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,946 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:58:06,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:06,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:06,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:06,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:58:06,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50a62e63[Count = 0] remaining members to acquire global barrier 2023-03-20 00:58:06,951 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:58:06,951 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,953 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,953 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:58:06,953 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:58:06,953 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:58:06,954 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,954 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:58:06,955 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,955 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:58:06,955 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:06,955 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:58:06,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:06,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:06,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:06,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:06,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:06,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:58:06,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:58:06,959 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:58:06,959 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:58:06,959 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:58:06,959 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:58:06,959 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:58:06,961 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,961 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,962 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:06,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:06,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,962 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:06,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:06,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:06,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:06,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,965 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:06,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:06,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:06,968 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:06,968 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:06,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:06,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:06,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:06,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:06,968 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,968 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,968 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:06,969 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:06,969 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,969 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:06,969 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:06,972 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:06,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:58:06,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:58:06,972 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:58:06,972 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:58:06,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:58:07,072 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:58:07,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:58:07,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:58:07,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:58:07,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:58:07,082 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:58:07,082 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:07,082 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:58:07,082 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:58:07,082 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,083 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:07,084 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:07,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:07,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:07,084 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:07,084 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,084 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:58:07,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:58:07,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:07,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:58:07,086 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:58:07,086 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:58:07,086 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:58:07,086 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:07,086 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:58:07,087 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:58:07,087 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:58:07,088 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:07,088 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:58:07,088 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:58:07,088 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:07,088 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:58:07,088 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:58:07,088 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,088 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:58:07,088 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:58:07,089 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:07,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:07,089 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:07,089 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,089 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:58:07,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:07,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:07,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:07,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:58:07,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@326f2836[Count = 0] remaining members to acquire global barrier 2023-03-20 00:58:07,093 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:58:07,093 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,095 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,095 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:58:07,098 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,098 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:58:07,098 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:58:07,098 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:58:07,100 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,100 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:58:07,100 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:07,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,100 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:58:07,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:07,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:07,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:07,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:07,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:07,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:07,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:07,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:58:07,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:58:07,104 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:58:07,104 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:58:07,104 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:58:07,105 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:58:07,105 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:58:07,107 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,107 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:07,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,107 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:07,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:07,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:07,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:07,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,108 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,109 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:07,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49480 deadline: 1679273947107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:07,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:07,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:07,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:07,113 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,113 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:07,113 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:07,113 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,113 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:07,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:07,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:58:07,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:58:07,113 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:07,113 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:07,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:07,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:07,113 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,114 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,114 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,114 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:58:07,114 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:58:07,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:58:07,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:07,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:07,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:07,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49468 deadline: 1679273947127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:07,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34553] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49438 deadline: 1679273947127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,132 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:07,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:49492 deadline: 1679273947128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,197 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/3e35b97f93c544e2b12e930c62930e20 2023-03-20 00:58:07,214 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:58:07,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:58:07,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:58:07,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:58:07,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:58:07,216 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:58:07,217 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:07,217 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:58:07,217 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:58:07,218 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,218 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:07,218 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,218 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:07,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:07,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:07,219 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,219 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:58:07,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:07,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:58:07,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:07,219 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:58:07,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:58:07,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:58:07,220 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:58:07,220 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:07,220 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:58:07,220 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:58:07,220 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:58:07,220 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:07,220 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:58:07,220 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:58:07,220 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:07,220 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:58:07,220 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:58:07,220 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,220 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:58:07,220 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:58:07,221 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,221 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:07,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:07,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:07,221 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,221 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:58:07,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:07,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:07,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:07,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:58:07,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41cd9f33[Count = 0] remaining members to acquire global barrier 2023-03-20 00:58:07,222 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:58:07,222 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,223 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,223 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:58:07,223 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:58:07,223 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:58:07,224 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,224 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:58:07,225 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:58:07,225 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:07,225 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:58:07,227 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:07,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:07,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:07,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:07,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:07,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:07,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:07,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:58:07,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:58:07,253 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:58:07,253 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:58:07,253 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:58:07,253 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:58:07,253 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:58:07,256 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,256 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:07,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,257 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:07,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:07,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:07,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:07,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:07,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:07,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,257 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:07,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:07,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,258 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:07,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:07,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,261 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,261 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:07,261 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:07,261 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,261 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:07,261 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:07,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:58:07,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:58:07,261 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,261 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,261 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:07,261 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:58:07,261 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:07,261 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:58:07,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:07,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:07,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:07,361 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:58:07,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:58:07,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:58:07,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:58:07,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:58:07,364 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:58:07,364 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:07,364 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:58:07,364 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:58:07,364 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,364 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:07,365 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,365 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:07,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:07,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:07,365 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,365 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:58:07,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:07,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:58:07,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:07,366 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:58:07,367 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:58:07,367 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:58:07,367 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:58:07,367 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:07,367 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:58:07,367 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:58:07,367 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:58:07,367 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:07,367 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:58:07,367 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:58:07,367 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:07,367 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:58:07,367 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:58:07,367 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,367 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:58:07,367 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:58:07,368 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,368 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:07,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:07,368 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,368 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:58:07,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:07,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:07,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:07,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:07,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:58:07,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b2efd8c[Count = 0] remaining members to acquire global barrier 2023-03-20 00:58:07,369 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:58:07,369 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,370 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,370 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,370 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:58:07,370 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:58:07,370 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:58:07,370 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:58:07,371 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,371 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:58:07,371 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:07,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:07,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:07,371 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:58:07,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:07,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:07,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:07,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:07,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:07,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:58:07,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:58:07,372 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:58:07,372 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:58:07,372 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:58:07,372 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:58:07,373 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:58:07,373 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,373 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:07,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:07,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,373 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:07,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:07,374 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:07,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:07,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:07,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:07,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:07,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,375 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:07,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:07,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,376 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:07,376 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:07,376 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:07,376 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:07,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:07,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:58:07,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:58:07,376 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:07,376 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:07,376 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,376 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:58:07,377 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:58:07,376 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,377 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,377 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:07,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:07,477 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:58:07,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:58:07,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:58:07,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:58:07,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:58:07,484 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:58:07,484 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:07,484 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:58:07,484 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:58:07,484 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,485 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:07,485 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:07,485 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:07,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:07,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:07,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:58:07,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:07,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:58:07,487 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,487 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:58:07,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:58:07,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:58:07,494 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:58:07,494 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:07,494 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:58:07,495 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:58:07,495 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:58:07,495 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:07,495 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:58:07,495 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:58:07,495 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:07,495 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:58:07,495 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:58:07,495 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,495 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:58:07,495 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:58:07,499 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,499 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:07,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:07,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:07,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:07,500 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,500 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:58:07,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:07,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:07,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:58:07,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a1a01bc[Count = 0] remaining members to acquire global barrier 2023-03-20 00:58:07,501 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:58:07,501 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,506 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,506 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:58:07,506 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:58:07,506 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:58:07,506 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,506 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:58:07,507 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,507 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:58:07,507 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:07,507 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:58:07,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:07,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:07,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:07,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:07,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:07,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:07,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:07,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:58:07,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:58:07,509 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:58:07,509 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:58:07,509 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:58:07,509 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:58:07,509 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:58:07,511 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,511 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,511 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:07,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:07,512 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:07,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:07,513 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:07,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:07,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:07,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,514 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,514 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:07,514 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:07,514 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,514 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:07,514 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:07,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:07,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:07,514 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:07,514 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:58:07,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:58:07,514 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:07,515 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:07,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:07,515 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:58:07,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:58:07,515 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:58:07,605 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/8f60acf3749042239395b6eb6b85a1ac 2023-03-20 00:58:07,608 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/55e51e67b9364e1babb8a9cec40707ae as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/55e51e67b9364e1babb8a9cec40707ae 2023-03-20 00:58:07,610 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/55e51e67b9364e1babb8a9cec40707ae, entries=400, sequenceid=367, filesize=59.8 K 2023-03-20 00:58:07,611 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/3e35b97f93c544e2b12e930c62930e20 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/3e35b97f93c544e2b12e930c62930e20 2023-03-20 00:58:07,613 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/3e35b97f93c544e2b12e930c62930e20, entries=150, sequenceid=367, filesize=11.9 K 2023-03-20 00:58:07,614 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/8f60acf3749042239395b6eb6b85a1ac as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/8f60acf3749042239395b6eb6b85a1ac 2023-03-20 00:58:07,615 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:58:07,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:58:07,616 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/8f60acf3749042239395b6eb6b85a1ac, entries=150, sequenceid=367, filesize=11.9 K 2023-03-20 00:58:07,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 00:58:07,617 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 3796a7d329c46cabe9e95c4c1fb73632 in 1657ms, sequenceid=367, compaction requested=true 2023-03-20 00:58:07,617 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:58:07,617 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:58:07,617 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-20 00:58:07,617 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-20 00:58:07,617 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:58:07,617 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:58:07,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 00:58:07,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 00:58:07,618 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 00:58:07,618 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:07,618 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 00:58:07,618 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 00:58:07,619 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,619 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:07,619 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73798 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-20 00:58:07,619 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 206250 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-20 00:58:07,619 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 3796a7d329c46cabe9e95c4c1fb73632/B is initiating minor compaction (all files) 2023-03-20 00:58:07,619 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 3796a7d329c46cabe9e95c4c1fb73632/A is initiating minor compaction (all files) 2023-03-20 00:58:07,619 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3796a7d329c46cabe9e95c4c1fb73632/B in TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:07,619 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3796a7d329c46cabe9e95c4c1fb73632/A in TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:07,619 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/9561670ffc9f4d809f8e5f968e699a69, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/052545d1c65f40098b5a1791a40e7abd, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/4970105f688d49a1b483d03b14c0c89f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/dafabe814e394d1b858104d29a7cd519, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/ef5d565fca1d4794b8b70762196e3ac1, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/3e35b97f93c544e2b12e930c62930e20] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp, totalSize=72.1 K 2023-03-20 00:58:07,619 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,619 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/0492140a091f42f8a467bef618498e1a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/2a080519b7604d4eb4e7203f64ac319e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/7ce59681697c47ad889430261153ac0b, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/a98b27075b00409f800d7fae992887be, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/711dc43500d94ff4a77a6c64cd410153, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/55e51e67b9364e1babb8a9cec40707ae] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp, totalSize=201.4 K 2023-03-20 00:58:07,619 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:07,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:07,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:07,620 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:07,620 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 00:58:07,620 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 9561670ffc9f4d809f8e5f968e699a69, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1679273882916 2023-03-20 00:58:07,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:07,620 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 0492140a091f42f8a467bef618498e1a, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1679273882916 2023-03-20 00:58:07,620 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 052545d1c65f40098b5a1791a40e7abd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1679273883035 2023-03-20 00:58:07,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:07,621 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 2a080519b7604d4eb4e7203f64ac319e, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1679273883035 2023-03-20 00:58:07,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 00:58:07,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:07,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 00:58:07,621 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 00:58:07,621 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 00:58:07,621 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 00:58:07,621 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 00:58:07,621 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 00:58:07,630 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 00:58:07,630 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:07,630 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. started... 2023-03-20 00:58:07,630 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 00:58:07,630 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3796a7d329c46cabe9e95c4c1fb73632 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-20 00:58:07,630 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 7ce59681697c47ad889430261153ac0b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1679273883160 2023-03-20 00:58:07,630 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 4970105f688d49a1b483d03b14c0c89f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1679273883160 2023-03-20 00:58:07,630 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=A 2023-03-20 00:58:07,631 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting a98b27075b00409f800d7fae992887be, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1679273883793 2023-03-20 00:58:07,631 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:58:07,631 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=B 2023-03-20 00:58:07,631 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting dafabe814e394d1b858104d29a7cd519, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1679273883793 2023-03-20 00:58:07,631 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 711dc43500d94ff4a77a6c64cd410153, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1679273884825 2023-03-20 00:58:07,631 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:58:07,631 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=C 2023-03-20 00:58:07,631 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 00:58:07,631 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 55e51e67b9364e1babb8a9cec40707ae, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1679273885935 2023-03-20 00:58:07,631 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting ef5d565fca1d4794b8b70762196e3ac1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1679273884825 2023-03-20 00:58:07,632 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 3e35b97f93c544e2b12e930c62930e20, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1679273885938 2023-03-20 00:58:07,645 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 3796a7d329c46cabe9e95c4c1fb73632#B#compaction#508 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:58:07,667 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3796a7d329c46cabe9e95c4c1fb73632#A#compaction#509 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:58:07,670 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320caf781ae4c71405d8997acd25dcb1bed to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320caf781ae4c71405d8997acd25dcb1bed 2023-03-20 00:58:07,675 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=402, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/a70d2a2313e548e1b828499d5097ccd3 2023-03-20 00:58:07,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:58:07,930 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. as already flushing 2023-03-20 00:58:07,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:07,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49504 deadline: 1679273947962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:08,008 DEBUG [Thread-3208] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7c00a18c to 127.0.0.1:60547 2023-03-20 00:58:08,008 DEBUG [Thread-3214] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70e63396 to 127.0.0.1:60547 2023-03-20 00:58:08,008 DEBUG [Thread-3208] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:58:08,008 DEBUG [Thread-3214] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:58:08,008 DEBUG [Thread-3210] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3da68fe0 to 127.0.0.1:60547 2023-03-20 00:58:08,008 DEBUG [Thread-3210] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:58:08,010 DEBUG [Thread-3212] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7fd90ac8 to 127.0.0.1:60547 2023-03-20 00:58:08,010 DEBUG [Thread-3212] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:58:08,013 DEBUG [Thread-3216] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x32cc4808 to 127.0.0.1:60547 2023-03-20 00:58:08,013 DEBUG [Thread-3216] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:58:08,068 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:08,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49504 deadline: 1679273948068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:08,070 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/531f68015a4541f8bda7cb269c5d2406 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/531f68015a4541f8bda7cb269c5d2406 2023-03-20 00:58:08,075 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 3796a7d329c46cabe9e95c4c1fb73632/B of 3796a7d329c46cabe9e95c4c1fb73632 into 531f68015a4541f8bda7cb269c5d2406(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:58:08,075 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:58:08,075 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632., storeName=3796a7d329c46cabe9e95c4c1fb73632/B, priority=10, startTime=1679273887617; duration=0sec 2023-03-20 00:58:08,075 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:58:08,076 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-20 00:58:08,080 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73798 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-20 00:58:08,080 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 3796a7d329c46cabe9e95c4c1fb73632/C is initiating minor compaction (all files) 2023-03-20 00:58:08,080 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3796a7d329c46cabe9e95c4c1fb73632/C in TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:08,080 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/8c59efd8d024430aa9fb6853e76e65bd, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/a8cd672775464b3da4f742b53023c90c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/19153e7eea48497986a8a7f97c8a8082, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/ce5963caedd1472d8a4f283ab550fdd0, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/a8364ac210ef48819bf3b4cb9b0fb127, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/8f60acf3749042239395b6eb6b85a1ac] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp, totalSize=72.1 K 2023-03-20 00:58:08,080 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 8c59efd8d024430aa9fb6853e76e65bd, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1679273882916 2023-03-20 00:58:08,081 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting a8cd672775464b3da4f742b53023c90c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1679273883035 2023-03-20 00:58:08,081 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 19153e7eea48497986a8a7f97c8a8082, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1679273883160 2023-03-20 00:58:08,081 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting ce5963caedd1472d8a4f283ab550fdd0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1679273883793 2023-03-20 00:58:08,081 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting a8364ac210ef48819bf3b4cb9b0fb127, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1679273884825 2023-03-20 00:58:08,082 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 8f60acf3749042239395b6eb6b85a1ac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1679273885938 2023-03-20 00:58:08,082 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/e0aa5774307042e5b802e1ef99d89916 2023-03-20 00:58:08,089 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 3796a7d329c46cabe9e95c4c1fb73632#C#compaction#512 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:58:08,103 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/ddb61fc9905e4e2693b10208fbefa925 2023-03-20 00:58:08,106 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/a70d2a2313e548e1b828499d5097ccd3 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/a70d2a2313e548e1b828499d5097ccd3 2023-03-20 00:58:08,110 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/a70d2a2313e548e1b828499d5097ccd3, entries=150, sequenceid=402, filesize=25.4 K 2023-03-20 00:58:08,111 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/e0aa5774307042e5b802e1ef99d89916 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/e0aa5774307042e5b802e1ef99d89916 2023-03-20 00:58:08,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 00:58:08,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:49480 deadline: 1679273948114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:08,114 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/e0aa5774307042e5b802e1ef99d89916, entries=150, sequenceid=402, filesize=11.9 K 2023-03-20 00:58:08,116 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/ddb61fc9905e4e2693b10208fbefa925 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/ddb61fc9905e4e2693b10208fbefa925 2023-03-20 00:58:08,119 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/ddb61fc9905e4e2693b10208fbefa925, entries=150, sequenceid=402, filesize=11.9 K 2023-03-20 00:58:08,120 INFO [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 3796a7d329c46cabe9e95c4c1fb73632 in 490ms, sequenceid=402, compaction requested=false 2023-03-20 00:58:08,120 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:58:08,120 DEBUG [rs(jenkins-hbase20.apache.org,34553,1679273725199)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:08,120 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 00:58:08,120 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 00:58:08,120 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:08,120 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 00:58:08,120 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 00:58:08,121 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:08,121 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:08,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:08,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:08,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:08,121 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:08,121 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 00:58:08,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:08,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:08,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:08,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:08,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:08,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34553,1679273725199' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 00:58:08,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51ffb451[Count = 0] remaining members to acquire global barrier 2023-03-20 00:58:08,122 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 00:58:08,122 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:08,123 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:08,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:08,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:08,123 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 00:58:08,123 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 00:58:08,123 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34553,1679273725199' in zk 2023-03-20 00:58:08,123 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:08,123 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 00:58:08,124 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:08,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:08,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:08,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:08,124 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 00:58:08,124 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:08,124 DEBUG [member: 'jenkins-hbase20.apache.org,34553,1679273725199' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 00:58:08,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:08,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:08,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:08,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:08,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:08,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:08,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:08,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34553,1679273725199': 2023-03-20 00:58:08,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34553,1679273725199' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 00:58:08,126 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 00:58:08,126 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 00:58:08,126 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 00:58:08,126 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 00:58:08,126 INFO [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 00:58:08,127 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:08,127 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:08,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:08,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:08,127 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:08,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:08,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 00:58:08,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 00:58:08,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:08,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:08,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:08,128 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:08,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 00:58:08,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:08,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 00:58:08,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:08,129 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:08,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:08,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 00:58:08,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 00:58:08,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:08,130 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:08,130 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 00:58:08,130 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:08,130 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 00:58:08,130 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:08,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 00:58:08,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 00:58:08,130 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 00:58:08,130 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:08,130 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 00:58:08,130 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 00:58:08,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 00:58:08,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 00:58:08,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 00:58:08,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 00:58:08,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 00:58:08,130 DEBUG [(jenkins-hbase20.apache.org,44395,1679273724282)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 00:58:08,130 DEBUG [Thread-3207] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 00:58:08,131 DEBUG [Thread-3207] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 00:58:08,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34553] regionserver.HRegion(9158): Flush requested on 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:58:08,132 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3796a7d329c46cabe9e95c4c1fb73632 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-20 00:58:08,132 DEBUG [Thread-3201] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4143b9ae to 127.0.0.1:60547 2023-03-20 00:58:08,132 DEBUG [Thread-3201] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:58:08,132 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=A 2023-03-20 00:58:08,132 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:58:08,132 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=B 2023-03-20 00:58:08,132 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:58:08,132 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=C 2023-03-20 00:58:08,132 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:58:08,136 DEBUG [Thread-3203] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x686af57e to 127.0.0.1:60547 2023-03-20 00:58:08,136 DEBUG [Thread-3203] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:58:08,138 DEBUG [Thread-3199] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3a1010a6 to 127.0.0.1:60547 2023-03-20 00:58:08,138 DEBUG [Thread-3199] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:58:08,231 DEBUG [Thread-3207] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 00:58:08,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 00:58:08,271 DEBUG [Thread-3197] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2f11e84a to 127.0.0.1:60547 2023-03-20 00:58:08,271 DEBUG [Thread-3197] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:58:08,509 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/2ab42d50899e4d55ad71ff08d6e7e4bb as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/2ab42d50899e4d55ad71ff08d6e7e4bb 2023-03-20 00:58:08,514 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/216ea1277afe4d7c90904d4b8c219200 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/216ea1277afe4d7c90904d4b8c219200 2023-03-20 00:58:08,518 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 3796a7d329c46cabe9e95c4c1fb73632/A of 3796a7d329c46cabe9e95c4c1fb73632 into 2ab42d50899e4d55ad71ff08d6e7e4bb(size=26.3 K), total size for store is 51.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:58:08,518 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:58:08,518 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632., storeName=3796a7d329c46cabe9e95c4c1fb73632/A, priority=10, startTime=1679273887617; duration=0sec 2023-03-20 00:58:08,518 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:58:08,522 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 3796a7d329c46cabe9e95c4c1fb73632/C of 3796a7d329c46cabe9e95c4c1fb73632 into 216ea1277afe4d7c90904d4b8c219200(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:58:08,522 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:58:08,522 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632., storeName=3796a7d329c46cabe9e95c4c1fb73632/C, priority=10, startTime=1679273887617; duration=0sec 2023-03-20 00:58:08,523 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:58:08,542 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320afc38176094849cdbef6064d8a1bc892 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320afc38176094849cdbef6064d8a1bc892 2023-03-20 00:58:08,952 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=415, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/c91e815e37f7451e990c46891f9b15db 2023-03-20 00:58:09,363 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/830199830af3488d9305d2d7600de934 2023-03-20 00:58:09,772 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/7c42b7d0eca0498ca7c6294350ba7112 2023-03-20 00:58:09,777 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/c91e815e37f7451e990c46891f9b15db as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/c91e815e37f7451e990c46891f9b15db 2023-03-20 00:58:09,781 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/c91e815e37f7451e990c46891f9b15db, entries=150, sequenceid=415, filesize=25.4 K 2023-03-20 00:58:09,782 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/830199830af3488d9305d2d7600de934 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/830199830af3488d9305d2d7600de934 2023-03-20 00:58:09,786 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/830199830af3488d9305d2d7600de934, entries=150, sequenceid=415, filesize=11.9 K 2023-03-20 00:58:09,787 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/7c42b7d0eca0498ca7c6294350ba7112 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/7c42b7d0eca0498ca7c6294350ba7112 2023-03-20 00:58:09,791 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/7c42b7d0eca0498ca7c6294350ba7112, entries=150, sequenceid=415, filesize=11.9 K 2023-03-20 00:58:09,791 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=20.13 KB/20610 for 3796a7d329c46cabe9e95c4c1fb73632 in 1659ms, sequenceid=415, compaction requested=true 2023-03-20 00:58:09,791 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:58:09,792 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:58:09,792 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:58:09,792 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:58:09,792 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:58:09,792 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:58:09,793 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 79024 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:58:09,793 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1912): 3796a7d329c46cabe9e95c4c1fb73632/A is initiating minor compaction (all files) 2023-03-20 00:58:09,793 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:58:09,793 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3796a7d329c46cabe9e95c4c1fb73632/A in TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:09,793 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 3796a7d329c46cabe9e95c4c1fb73632/B is initiating minor compaction (all files) 2023-03-20 00:58:09,793 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/2ab42d50899e4d55ad71ff08d6e7e4bb, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/a70d2a2313e548e1b828499d5097ccd3, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/c91e815e37f7451e990c46891f9b15db] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp, totalSize=77.2 K 2023-03-20 00:58:09,793 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3796a7d329c46cabe9e95c4c1fb73632/B in TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:09,793 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/531f68015a4541f8bda7cb269c5d2406, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/e0aa5774307042e5b802e1ef99d89916, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/830199830af3488d9305d2d7600de934] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp, totalSize=36.6 K 2023-03-20 00:58:09,793 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting 2ab42d50899e4d55ad71ff08d6e7e4bb, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1679273885938 2023-03-20 00:58:09,794 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 531f68015a4541f8bda7cb269c5d2406, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1679273885938 2023-03-20 00:58:09,794 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting a70d2a2313e548e1b828499d5097ccd3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1679273886005 2023-03-20 00:58:09,794 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting e0aa5774307042e5b802e1ef99d89916, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1679273886005 2023-03-20 00:58:09,794 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] compactions.Compactor(207): Compacting c91e815e37f7451e990c46891f9b15db, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1679273887937 2023-03-20 00:58:09,794 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 830199830af3488d9305d2d7600de934, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1679273887937 2023-03-20 00:58:09,809 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 3796a7d329c46cabe9e95c4c1fb73632#B#compaction#517 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:58:09,814 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3796a7d329c46cabe9e95c4c1fb73632#A#compaction#516 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:58:09,821 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/39e640be5fe44b328f11455cc31eb26b as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/39e640be5fe44b328f11455cc31eb26b 2023-03-20 00:58:09,825 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3796a7d329c46cabe9e95c4c1fb73632/B of 3796a7d329c46cabe9e95c4c1fb73632 into 39e640be5fe44b328f11455cc31eb26b(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:58:09,825 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:58:09,825 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632., storeName=3796a7d329c46cabe9e95c4c1fb73632/B, priority=13, startTime=1679273889792; duration=0sec 2023-03-20 00:58:09,826 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 00:58:09,826 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 00:58:09,826 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 00:58:09,827 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1912): 3796a7d329c46cabe9e95c4c1fb73632/C is initiating minor compaction (all files) 2023-03-20 00:58:09,827 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3796a7d329c46cabe9e95c4c1fb73632/C in TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:09,827 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/216ea1277afe4d7c90904d4b8c219200, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/ddb61fc9905e4e2693b10208fbefa925, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/7c42b7d0eca0498ca7c6294350ba7112] into tmpdir=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp, totalSize=36.6 K 2023-03-20 00:58:09,827 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 216ea1277afe4d7c90904d4b8c219200, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1679273885938 2023-03-20 00:58:09,827 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting ddb61fc9905e4e2693b10208fbefa925, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1679273886005 2023-03-20 00:58:09,828 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] compactions.Compactor(207): Compacting 7c42b7d0eca0498ca7c6294350ba7112, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1679273887937 2023-03-20 00:58:09,834 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] throttle.PressureAwareThroughputController(145): 3796a7d329c46cabe9e95c4c1fb73632#C#compaction#518 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 00:58:10,126 DEBUG [Thread-3205] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x078cd331 to 127.0.0.1:60547 2023-03-20 00:58:10,126 DEBUG [Thread-3205] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:58:10,126 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-20 00:58:10,127 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(394): wrote 78 2023-03-20 00:58:10,127 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(394): wrote 47 2023-03-20 00:58:10,127 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(394): wrote 72 2023-03-20 00:58:10,127 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(394): wrote 68 2023-03-20 00:58:10,127 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(394): wrote 65 2023-03-20 00:58:10,127 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-20 00:58:10,127 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(398): read 5487 2023-03-20 00:58:10,127 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(398): read 5497 2023-03-20 00:58:10,127 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(398): read 5342 2023-03-20 00:58:10,127 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(398): read 5465 2023-03-20 00:58:10,127 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(398): read 5476 2023-03-20 00:58:10,127 INFO [Listener at localhost.localdomain/43361] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-20 00:58:10,128 INFO [Listener at localhost.localdomain/43361] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-03-20 00:58:10,128 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5be55982 to 127.0.0.1:60547 2023-03-20 00:58:10,128 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:58:10,128 INFO [Listener at localhost.localdomain/43361] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-20 00:58:10,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-03-20 00:58:10,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-20 00:58:10,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-20 00:58:10,132 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679273890132"}]},"ts":"1679273890132"} 2023-03-20 00:58:10,133 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-20 00:58:10,135 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-20 00:58:10,135 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3796a7d329c46cabe9e95c4c1fb73632, UNASSIGN}] 2023-03-20 00:58:10,137 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3796a7d329c46cabe9e95c4c1fb73632, UNASSIGN 2023-03-20 00:58:10,137 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=3796a7d329c46cabe9e95c4c1fb73632, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:10,138 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273890137"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679273890137"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679273890137"}]},"ts":"1679273890137"} 2023-03-20 00:58:10,139 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199}] 2023-03-20 00:58:10,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-20 00:58:10,244 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/7ffa2ffa15ac421299c9bdbc051ab81a as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/7ffa2ffa15ac421299c9bdbc051ab81a 2023-03-20 00:58:10,248 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/e9c6b61766a148138ea143e72d68c2b9 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/e9c6b61766a148138ea143e72d68c2b9 2023-03-20 00:58:10,252 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3796a7d329c46cabe9e95c4c1fb73632/A of 3796a7d329c46cabe9e95c4c1fb73632 into 7ffa2ffa15ac421299c9bdbc051ab81a(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:58:10,252 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:58:10,252 INFO [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632., storeName=3796a7d329c46cabe9e95c4c1fb73632/A, priority=13, startTime=1679273889792; duration=0sec 2023-03-20 00:58:10,252 DEBUG [RS:0;jenkins-hbase20:34553-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:58:10,254 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3796a7d329c46cabe9e95c4c1fb73632/C of 3796a7d329c46cabe9e95c4c1fb73632 into e9c6b61766a148138ea143e72d68c2b9(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 00:58:10,254 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:58:10,254 INFO [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632., storeName=3796a7d329c46cabe9e95c4c1fb73632/C, priority=13, startTime=1679273889792; duration=0sec 2023-03-20 00:58:10,254 DEBUG [RS:0;jenkins-hbase20:34553-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 00:58:10,291 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:58:10,291 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 3796a7d329c46cabe9e95c4c1fb73632, disabling compactions & flushes 2023-03-20 00:58:10,291 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:10,291 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:10,291 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. after waiting 0 ms 2023-03-20 00:58:10,291 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:10,291 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 3796a7d329c46cabe9e95c4c1fb73632 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-20 00:58:10,292 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=A 2023-03-20 00:58:10,292 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:58:10,292 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=B 2023-03-20 00:58:10,292 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:58:10,292 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3796a7d329c46cabe9e95c4c1fb73632, store=C 2023-03-20 00:58:10,292 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 00:58:10,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-20 00:58:10,712 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303206af0921b344f44e28bd31d05fe59fbb5 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303206af0921b344f44e28bd31d05fe59fbb5 2023-03-20 00:58:10,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-20 00:58:11,118 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=427, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/9f90ca4c18d740e2b2448f656b763cc9 2023-03-20 00:58:11,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-20 00:58:11,526 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/d499563ee6ae4a53bb249923fb39a3d3 2023-03-20 00:58:11,542 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/a2537415c61443fa8185a98b4467167e 2023-03-20 00:58:11,547 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/A/9f90ca4c18d740e2b2448f656b763cc9 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/9f90ca4c18d740e2b2448f656b763cc9 2023-03-20 00:58:11,551 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/9f90ca4c18d740e2b2448f656b763cc9, entries=150, sequenceid=427, filesize=25.4 K 2023-03-20 00:58:11,552 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/B/d499563ee6ae4a53bb249923fb39a3d3 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/d499563ee6ae4a53bb249923fb39a3d3 2023-03-20 00:58:11,555 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/d499563ee6ae4a53bb249923fb39a3d3, entries=150, sequenceid=427, filesize=11.9 K 2023-03-20 00:58:11,555 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/.tmp/C/a2537415c61443fa8185a98b4467167e as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/a2537415c61443fa8185a98b4467167e 2023-03-20 00:58:11,558 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/a2537415c61443fa8185a98b4467167e, entries=150, sequenceid=427, filesize=11.9 K 2023-03-20 00:58:11,558 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 3796a7d329c46cabe9e95c4c1fb73632 in 1267ms, sequenceid=427, compaction requested=false 2023-03-20 00:58:11,559 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/606efe733faf45cf99fefcc8449533fa, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/930d42d43f8045c7b8c7594b5bb43859, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/039fe2158935405da3d7dbca3b311a23, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/a338c35e8eca47eaa7bac41675cabf3a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/a2fa67cc064a46479dd1e32e47148b93, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/b2deaba938f341389aa0dd3adfc87118, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/3840728968a048a48551609b8faefa54, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/3b5671602e0f4e2085a1f33047f9a314, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/90245c25b766488c90e61b9bb56ce5ee, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/0a6edebdbc0449f897117f8d8d7e0d14, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/1f9ee2f684ea4912809cdffde6c8a42f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/aa0f834a18394985ab007c2a59e7db3a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/182bb2d36651472cb12b73a0fc2409ed, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/9cc7db9907e14fa3bccbe944df38e00a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/beffb6edf5934c6bb6d05877f9606e6a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/6fb41291e899495ba3e9528292d314d6, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/7c55abecca264e8baea6b216a06d4a0d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/8774e4bf7ab344b5a31a9f9a26264cf3, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/a7b0d0b1319341b4ab910231ad6f710e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/0492140a091f42f8a467bef618498e1a, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/1b55b314dc8348e4bc8a2fac726c29d4, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/2a080519b7604d4eb4e7203f64ac319e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/7ce59681697c47ad889430261153ac0b, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/a98b27075b00409f800d7fae992887be, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/711dc43500d94ff4a77a6c64cd410153, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/55e51e67b9364e1babb8a9cec40707ae, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/2ab42d50899e4d55ad71ff08d6e7e4bb, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/a70d2a2313e548e1b828499d5097ccd3, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/c91e815e37f7451e990c46891f9b15db] to archive 2023-03-20 00:58:11,560 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 00:58:11,562 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/606efe733faf45cf99fefcc8449533fa to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/606efe733faf45cf99fefcc8449533fa 2023-03-20 00:58:11,563 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/930d42d43f8045c7b8c7594b5bb43859 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/930d42d43f8045c7b8c7594b5bb43859 2023-03-20 00:58:11,564 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/039fe2158935405da3d7dbca3b311a23 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/039fe2158935405da3d7dbca3b311a23 2023-03-20 00:58:11,565 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/a338c35e8eca47eaa7bac41675cabf3a to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/a338c35e8eca47eaa7bac41675cabf3a 2023-03-20 00:58:11,565 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/a2fa67cc064a46479dd1e32e47148b93 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/a2fa67cc064a46479dd1e32e47148b93 2023-03-20 00:58:11,566 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/b2deaba938f341389aa0dd3adfc87118 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/b2deaba938f341389aa0dd3adfc87118 2023-03-20 00:58:11,567 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/3840728968a048a48551609b8faefa54 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/3840728968a048a48551609b8faefa54 2023-03-20 00:58:11,568 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/3b5671602e0f4e2085a1f33047f9a314 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/3b5671602e0f4e2085a1f33047f9a314 2023-03-20 00:58:11,569 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/90245c25b766488c90e61b9bb56ce5ee to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/90245c25b766488c90e61b9bb56ce5ee 2023-03-20 00:58:11,570 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/0a6edebdbc0449f897117f8d8d7e0d14 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/0a6edebdbc0449f897117f8d8d7e0d14 2023-03-20 00:58:11,571 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/1f9ee2f684ea4912809cdffde6c8a42f to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/1f9ee2f684ea4912809cdffde6c8a42f 2023-03-20 00:58:11,571 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/aa0f834a18394985ab007c2a59e7db3a to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/aa0f834a18394985ab007c2a59e7db3a 2023-03-20 00:58:11,572 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/182bb2d36651472cb12b73a0fc2409ed to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/182bb2d36651472cb12b73a0fc2409ed 2023-03-20 00:58:11,573 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/9cc7db9907e14fa3bccbe944df38e00a to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/9cc7db9907e14fa3bccbe944df38e00a 2023-03-20 00:58:11,574 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/beffb6edf5934c6bb6d05877f9606e6a to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/beffb6edf5934c6bb6d05877f9606e6a 2023-03-20 00:58:11,575 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/6fb41291e899495ba3e9528292d314d6 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/6fb41291e899495ba3e9528292d314d6 2023-03-20 00:58:11,575 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/7c55abecca264e8baea6b216a06d4a0d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/7c55abecca264e8baea6b216a06d4a0d 2023-03-20 00:58:11,576 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/8774e4bf7ab344b5a31a9f9a26264cf3 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/8774e4bf7ab344b5a31a9f9a26264cf3 2023-03-20 00:58:11,577 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/a7b0d0b1319341b4ab910231ad6f710e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/a7b0d0b1319341b4ab910231ad6f710e 2023-03-20 00:58:11,578 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/0492140a091f42f8a467bef618498e1a to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/0492140a091f42f8a467bef618498e1a 2023-03-20 00:58:11,578 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/1b55b314dc8348e4bc8a2fac726c29d4 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/1b55b314dc8348e4bc8a2fac726c29d4 2023-03-20 00:58:11,579 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/2a080519b7604d4eb4e7203f64ac319e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/2a080519b7604d4eb4e7203f64ac319e 2023-03-20 00:58:11,579 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/7ce59681697c47ad889430261153ac0b to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/7ce59681697c47ad889430261153ac0b 2023-03-20 00:58:11,580 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/a98b27075b00409f800d7fae992887be to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/a98b27075b00409f800d7fae992887be 2023-03-20 00:58:11,581 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/711dc43500d94ff4a77a6c64cd410153 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/711dc43500d94ff4a77a6c64cd410153 2023-03-20 00:58:11,581 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/55e51e67b9364e1babb8a9cec40707ae to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/55e51e67b9364e1babb8a9cec40707ae 2023-03-20 00:58:11,582 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/2ab42d50899e4d55ad71ff08d6e7e4bb to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/2ab42d50899e4d55ad71ff08d6e7e4bb 2023-03-20 00:58:11,582 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/a70d2a2313e548e1b828499d5097ccd3 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/a70d2a2313e548e1b828499d5097ccd3 2023-03-20 00:58:11,583 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/c91e815e37f7451e990c46891f9b15db to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/c91e815e37f7451e990c46891f9b15db 2023-03-20 00:58:11,585 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/3900c55c86984042ae1c9b7ace407527, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/46c0d7ada6bb4f9d81455536b8fe5a75, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/eaa49ece820b4bcfbeb9833fb1469718, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/737986ec6d11401d9b89fd8fc078f805, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/191ed921d07b4d029528a1eddde6c05d, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/a6573eb527d0450abb9e4d226fad778f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/a16304ef2a28409a8687e7d75b4d9fc0, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/58b88b04c13e4d0297532175f26249fe, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/3d3bb2215bdf44ae8fcc516c9fabd718, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/53c7f5aca1c140348d2c3d4b1189e0cc, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/aa1ce43eb5fb432f92c3c4ebdc6c96b0, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/c6fc8cf8ab5f47f98c005092098c757b, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/5f5cef1394344482b8b67bc172054b55, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/85d763ed2a5442389a68ece12bd82042, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/588cdb929d494cb783e589e32f07a91f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/10f125b3bfac4ab7a62bf399d68160e7, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/21038d6bd917471a97dbec7f894f70e9, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/2159ddd05afe45eeb4dd9a5dc0e9e9ff, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/0ffb5b804cd749b2aa0bc96fde9bdb6b, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/9561670ffc9f4d809f8e5f968e699a69, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/37d9f1d52796408a9d168ee30ba2c641, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/052545d1c65f40098b5a1791a40e7abd, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/4970105f688d49a1b483d03b14c0c89f, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/dafabe814e394d1b858104d29a7cd519, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/ef5d565fca1d4794b8b70762196e3ac1, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/531f68015a4541f8bda7cb269c5d2406, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/3e35b97f93c544e2b12e930c62930e20, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/e0aa5774307042e5b802e1ef99d89916, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/830199830af3488d9305d2d7600de934] to archive 2023-03-20 00:58:11,586 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 00:58:11,587 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/3900c55c86984042ae1c9b7ace407527 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/3900c55c86984042ae1c9b7ace407527 2023-03-20 00:58:11,588 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/46c0d7ada6bb4f9d81455536b8fe5a75 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/46c0d7ada6bb4f9d81455536b8fe5a75 2023-03-20 00:58:11,588 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/eaa49ece820b4bcfbeb9833fb1469718 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/eaa49ece820b4bcfbeb9833fb1469718 2023-03-20 00:58:11,589 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/737986ec6d11401d9b89fd8fc078f805 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/737986ec6d11401d9b89fd8fc078f805 2023-03-20 00:58:11,589 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/191ed921d07b4d029528a1eddde6c05d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/191ed921d07b4d029528a1eddde6c05d 2023-03-20 00:58:11,590 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/a6573eb527d0450abb9e4d226fad778f to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/a6573eb527d0450abb9e4d226fad778f 2023-03-20 00:58:11,591 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/a16304ef2a28409a8687e7d75b4d9fc0 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/a16304ef2a28409a8687e7d75b4d9fc0 2023-03-20 00:58:11,591 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/58b88b04c13e4d0297532175f26249fe to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/58b88b04c13e4d0297532175f26249fe 2023-03-20 00:58:11,593 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/3d3bb2215bdf44ae8fcc516c9fabd718 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/3d3bb2215bdf44ae8fcc516c9fabd718 2023-03-20 00:58:11,593 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/53c7f5aca1c140348d2c3d4b1189e0cc to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/53c7f5aca1c140348d2c3d4b1189e0cc 2023-03-20 00:58:11,594 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/aa1ce43eb5fb432f92c3c4ebdc6c96b0 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/aa1ce43eb5fb432f92c3c4ebdc6c96b0 2023-03-20 00:58:11,595 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/c6fc8cf8ab5f47f98c005092098c757b to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/c6fc8cf8ab5f47f98c005092098c757b 2023-03-20 00:58:11,596 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/5f5cef1394344482b8b67bc172054b55 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/5f5cef1394344482b8b67bc172054b55 2023-03-20 00:58:11,597 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/85d763ed2a5442389a68ece12bd82042 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/85d763ed2a5442389a68ece12bd82042 2023-03-20 00:58:11,597 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/588cdb929d494cb783e589e32f07a91f to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/588cdb929d494cb783e589e32f07a91f 2023-03-20 00:58:11,598 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/10f125b3bfac4ab7a62bf399d68160e7 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/10f125b3bfac4ab7a62bf399d68160e7 2023-03-20 00:58:11,599 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/21038d6bd917471a97dbec7f894f70e9 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/21038d6bd917471a97dbec7f894f70e9 2023-03-20 00:58:11,600 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/2159ddd05afe45eeb4dd9a5dc0e9e9ff to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/2159ddd05afe45eeb4dd9a5dc0e9e9ff 2023-03-20 00:58:11,601 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/0ffb5b804cd749b2aa0bc96fde9bdb6b to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/0ffb5b804cd749b2aa0bc96fde9bdb6b 2023-03-20 00:58:11,601 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/9561670ffc9f4d809f8e5f968e699a69 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/9561670ffc9f4d809f8e5f968e699a69 2023-03-20 00:58:11,602 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/37d9f1d52796408a9d168ee30ba2c641 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/37d9f1d52796408a9d168ee30ba2c641 2023-03-20 00:58:11,603 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/052545d1c65f40098b5a1791a40e7abd to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/052545d1c65f40098b5a1791a40e7abd 2023-03-20 00:58:11,604 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/4970105f688d49a1b483d03b14c0c89f to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/4970105f688d49a1b483d03b14c0c89f 2023-03-20 00:58:11,605 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/dafabe814e394d1b858104d29a7cd519 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/dafabe814e394d1b858104d29a7cd519 2023-03-20 00:58:11,607 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/ef5d565fca1d4794b8b70762196e3ac1 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/ef5d565fca1d4794b8b70762196e3ac1 2023-03-20 00:58:11,609 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/531f68015a4541f8bda7cb269c5d2406 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/531f68015a4541f8bda7cb269c5d2406 2023-03-20 00:58:11,611 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/3e35b97f93c544e2b12e930c62930e20 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/3e35b97f93c544e2b12e930c62930e20 2023-03-20 00:58:11,612 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/e0aa5774307042e5b802e1ef99d89916 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/e0aa5774307042e5b802e1ef99d89916 2023-03-20 00:58:11,613 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/830199830af3488d9305d2d7600de934 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/830199830af3488d9305d2d7600de934 2023-03-20 00:58:11,617 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/04b5de4c1b2e40749030f64ed7e0ba39, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/b787a30d2a344679a3f6a8b89d5075cb, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/6664106bca4548f2b3926ca0f1844b4c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/0602bb8f68e24aba88bad0b1c9f0e1f3, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/9b6989ea5c6041f9b3fa766883c7070e, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/50a056a76cf7426d8b56ec16ec817966, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/8c104384611e496faf969cc401ee6679, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/589b0ba7b3914b5e8545f75474ce2fd4, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/53b146ebebaf4e7696f9e50f5c9b36b5, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/0ab92f8a8ed34e2690a032c45b6b8b96, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/c299cf5972cd4d0fbc4f101e64f882c9, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/823292529b944ad9ba9b5022d9b45102, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/a49053a121a64085a103be1cbfc02256, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/cd716c731fbe41db8e52aaa749112c0c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/50c4354bdf024447864015a72e2007b6, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/393764122b3a411592c9abe3c2ef6874, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/131832804e174057880c4a023d4499fa, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/07b1a309cab44ba0a143e422058a01d9, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/1fc0ede45fcd444ca9c1adc2c3ec9b7c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/8c59efd8d024430aa9fb6853e76e65bd, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/1a595520a8f64dac8d19c35b4024ea46, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/a8cd672775464b3da4f742b53023c90c, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/19153e7eea48497986a8a7f97c8a8082, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/ce5963caedd1472d8a4f283ab550fdd0, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/a8364ac210ef48819bf3b4cb9b0fb127, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/216ea1277afe4d7c90904d4b8c219200, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/8f60acf3749042239395b6eb6b85a1ac, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/ddb61fc9905e4e2693b10208fbefa925, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/7c42b7d0eca0498ca7c6294350ba7112] to archive 2023-03-20 00:58:11,618 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 00:58:11,620 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/04b5de4c1b2e40749030f64ed7e0ba39 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/04b5de4c1b2e40749030f64ed7e0ba39 2023-03-20 00:58:11,621 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/b787a30d2a344679a3f6a8b89d5075cb to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/b787a30d2a344679a3f6a8b89d5075cb 2023-03-20 00:58:11,623 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/6664106bca4548f2b3926ca0f1844b4c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/6664106bca4548f2b3926ca0f1844b4c 2023-03-20 00:58:11,624 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/0602bb8f68e24aba88bad0b1c9f0e1f3 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/0602bb8f68e24aba88bad0b1c9f0e1f3 2023-03-20 00:58:11,625 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/9b6989ea5c6041f9b3fa766883c7070e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/9b6989ea5c6041f9b3fa766883c7070e 2023-03-20 00:58:11,625 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/50a056a76cf7426d8b56ec16ec817966 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/50a056a76cf7426d8b56ec16ec817966 2023-03-20 00:58:11,626 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/8c104384611e496faf969cc401ee6679 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/8c104384611e496faf969cc401ee6679 2023-03-20 00:58:11,627 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/589b0ba7b3914b5e8545f75474ce2fd4 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/589b0ba7b3914b5e8545f75474ce2fd4 2023-03-20 00:58:11,628 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/53b146ebebaf4e7696f9e50f5c9b36b5 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/53b146ebebaf4e7696f9e50f5c9b36b5 2023-03-20 00:58:11,628 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/0ab92f8a8ed34e2690a032c45b6b8b96 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/0ab92f8a8ed34e2690a032c45b6b8b96 2023-03-20 00:58:11,629 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/c299cf5972cd4d0fbc4f101e64f882c9 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/c299cf5972cd4d0fbc4f101e64f882c9 2023-03-20 00:58:11,630 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/823292529b944ad9ba9b5022d9b45102 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/823292529b944ad9ba9b5022d9b45102 2023-03-20 00:58:11,631 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/a49053a121a64085a103be1cbfc02256 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/a49053a121a64085a103be1cbfc02256 2023-03-20 00:58:11,631 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/cd716c731fbe41db8e52aaa749112c0c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/cd716c731fbe41db8e52aaa749112c0c 2023-03-20 00:58:11,632 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/50c4354bdf024447864015a72e2007b6 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/50c4354bdf024447864015a72e2007b6 2023-03-20 00:58:11,633 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/393764122b3a411592c9abe3c2ef6874 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/393764122b3a411592c9abe3c2ef6874 2023-03-20 00:58:11,634 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/131832804e174057880c4a023d4499fa to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/131832804e174057880c4a023d4499fa 2023-03-20 00:58:11,634 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/07b1a309cab44ba0a143e422058a01d9 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/07b1a309cab44ba0a143e422058a01d9 2023-03-20 00:58:11,635 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/1fc0ede45fcd444ca9c1adc2c3ec9b7c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/1fc0ede45fcd444ca9c1adc2c3ec9b7c 2023-03-20 00:58:11,636 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/8c59efd8d024430aa9fb6853e76e65bd to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/8c59efd8d024430aa9fb6853e76e65bd 2023-03-20 00:58:11,636 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/1a595520a8f64dac8d19c35b4024ea46 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/1a595520a8f64dac8d19c35b4024ea46 2023-03-20 00:58:11,637 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/a8cd672775464b3da4f742b53023c90c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/a8cd672775464b3da4f742b53023c90c 2023-03-20 00:58:11,638 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/19153e7eea48497986a8a7f97c8a8082 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/19153e7eea48497986a8a7f97c8a8082 2023-03-20 00:58:11,639 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/ce5963caedd1472d8a4f283ab550fdd0 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/ce5963caedd1472d8a4f283ab550fdd0 2023-03-20 00:58:11,639 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/a8364ac210ef48819bf3b4cb9b0fb127 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/a8364ac210ef48819bf3b4cb9b0fb127 2023-03-20 00:58:11,640 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/216ea1277afe4d7c90904d4b8c219200 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/216ea1277afe4d7c90904d4b8c219200 2023-03-20 00:58:11,641 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/8f60acf3749042239395b6eb6b85a1ac to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/8f60acf3749042239395b6eb6b85a1ac 2023-03-20 00:58:11,641 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/ddb61fc9905e4e2693b10208fbefa925 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/ddb61fc9905e4e2693b10208fbefa925 2023-03-20 00:58:11,642 DEBUG [StoreCloser-TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/7c42b7d0eca0498ca7c6294350ba7112 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/7c42b7d0eca0498ca7c6294350ba7112 2023-03-20 00:58:11,645 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/recovered.edits/430.seqid, newMaxSeqId=430, maxSeqId=4 2023-03-20 00:58:11,646 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632. 2023-03-20 00:58:11,646 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 3796a7d329c46cabe9e95c4c1fb73632: 2023-03-20 00:58:11,647 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:58:11,647 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=3796a7d329c46cabe9e95c4c1fb73632, regionState=CLOSED 2023-03-20 00:58:11,647 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679273891647"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679273891647"}]},"ts":"1679273891647"} 2023-03-20 00:58:11,649 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-03-20 00:58:11,649 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 3796a7d329c46cabe9e95c4c1fb73632, server=jenkins-hbase20.apache.org,34553,1679273725199 in 1.5090 sec 2023-03-20 00:58:11,650 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-03-20 00:58:11,650 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3796a7d329c46cabe9e95c4c1fb73632, UNASSIGN in 1.5140 sec 2023-03-20 00:58:11,650 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679273891650"}]},"ts":"1679273891650"} 2023-03-20 00:58:11,651 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-20 00:58:11,652 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-20 00:58:11,653 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5230 sec 2023-03-20 00:58:12,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-20 00:58:12,241 INFO [Listener at localhost.localdomain/43361] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-03-20 00:58:12,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-03-20 00:58:12,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:58:12,244 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:58:12,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-20 00:58:12,244 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:58:12,251 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:58:12,253 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A, FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B, FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C, FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/recovered.edits] 2023-03-20 00:58:12,255 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/7ffa2ffa15ac421299c9bdbc051ab81a to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/7ffa2ffa15ac421299c9bdbc051ab81a 2023-03-20 00:58:12,256 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/9f90ca4c18d740e2b2448f656b763cc9 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/A/9f90ca4c18d740e2b2448f656b763cc9 2023-03-20 00:58:12,258 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/39e640be5fe44b328f11455cc31eb26b to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/39e640be5fe44b328f11455cc31eb26b 2023-03-20 00:58:12,259 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/d499563ee6ae4a53bb249923fb39a3d3 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/B/d499563ee6ae4a53bb249923fb39a3d3 2023-03-20 00:58:12,261 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/a2537415c61443fa8185a98b4467167e to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/a2537415c61443fa8185a98b4467167e 2023-03-20 00:58:12,261 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/e9c6b61766a148138ea143e72d68c2b9 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/C/e9c6b61766a148138ea143e72d68c2b9 2023-03-20 00:58:12,263 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/recovered.edits/430.seqid to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632/recovered.edits/430.seqid 2023-03-20 00:58:12,263 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/.tmp/data/default/TestAcidGuarantees/3796a7d329c46cabe9e95c4c1fb73632 2023-03-20 00:58:12,263 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-20 00:58:12,264 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-20 00:58:12,264 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-20 00:58:12,265 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303201637260ae06e4ee9a6bc0ded3f612226 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303201637260ae06e4ee9a6bc0ded3f612226 2023-03-20 00:58:12,266 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303201a24d452d9614a85bd782b9f0bd391b5 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303201a24d452d9614a85bd782b9f0bd391b5 2023-03-20 00:58:12,267 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032036687c9f7ebb4d0fbc47dc0343242352 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032036687c9f7ebb4d0fbc47dc0343242352 2023-03-20 00:58:12,267 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032037584bd6cccf430db0a603b95e0e3d17 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032037584bd6cccf430db0a603b95e0e3d17 2023-03-20 00:58:12,268 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032040d676774d964e58941f1d08f2bd7dd0 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032040d676774d964e58941f1d08f2bd7dd0 2023-03-20 00:58:12,269 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320556fcd6af73a4b24a0d3b7e84d6f1fc7 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320556fcd6af73a4b24a0d3b7e84d6f1fc7 2023-03-20 00:58:12,269 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303205af86cb4155346b98e6270fc595b4440 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303205af86cb4155346b98e6270fc595b4440 2023-03-20 00:58:12,270 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032060a90b2aaaf04c348baf1fc954dd5e25 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032060a90b2aaaf04c348baf1fc954dd5e25 2023-03-20 00:58:12,271 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303206af0921b344f44e28bd31d05fe59fbb5 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303206af0921b344f44e28bd31d05fe59fbb5 2023-03-20 00:58:12,271 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032084a583b4f4c64f2198ba63a7da3b878d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032084a583b4f4c64f2198ba63a7da3b878d 2023-03-20 00:58:12,272 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303208da6b2bba13e428e9499fd8e043377c2 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303208da6b2bba13e428e9499fd8e043377c2 2023-03-20 00:58:12,273 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032093534997b7524e4a805ed6d9acd2fb39 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032093534997b7524e4a805ed6d9acd2fb39 2023-03-20 00:58:12,273 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032094447f45f0e74c679a36b94223cb2cde to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032094447f45f0e74c679a36b94223cb2cde 2023-03-20 00:58:12,274 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303209699294374a2442d8dd3fb78f83e911d to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303209699294374a2442d8dd3fb78f83e911d 2023-03-20 00:58:12,275 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320afc38176094849cdbef6064d8a1bc892 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320afc38176094849cdbef6064d8a1bc892 2023-03-20 00:58:12,275 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320caf781ae4c71405d8997acd25dcb1bed to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320caf781ae4c71405d8997acd25dcb1bed 2023-03-20 00:58:12,276 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320cd3b493dce6b45c8a345bd8ece5d66dd to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320cd3b493dce6b45c8a345bd8ece5d66dd 2023-03-20 00:58:12,277 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320db3fff6821ed4c819c9af33d92e8bf84 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320db3fff6821ed4c819c9af33d92e8bf84 2023-03-20 00:58:12,277 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320e1ffcdbffc154535937ed329d64dce30 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320e1ffcdbffc154535937ed329d64dce30 2023-03-20 00:58:12,278 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320e50f299287004606852580a66af9dd3c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320e50f299287004606852580a66af9dd3c 2023-03-20 00:58:12,279 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320ed43d0feb09240c09f92507bf7094480 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320ed43d0feb09240c09f92507bf7094480 2023-03-20 00:58:12,279 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320f223a7b1230c460baae5d5985547b36c to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320f223a7b1230c460baae5d5985547b36c 2023-03-20 00:58:12,280 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320fa175949f1f24fd9b6babda5bb07e262 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320fa175949f1f24fd9b6babda5bb07e262 2023-03-20 00:58:12,281 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320feb82ee0544a492798a0a3290bf0d265 to hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320feb82ee0544a492798a0a3290bf0d265 2023-03-20 00:58:12,281 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-20 00:58:12,282 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:58:12,283 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-20 00:58:12,284 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-20 00:58:12,285 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:58:12,285 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-20 00:58:12,285 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679273892285"}]},"ts":"9223372036854775807"} 2023-03-20 00:58:12,286 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-20 00:58:12,286 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 3796a7d329c46cabe9e95c4c1fb73632, NAME => 'TestAcidGuarantees,,1679273865228.3796a7d329c46cabe9e95c4c1fb73632.', STARTKEY => '', ENDKEY => ''}] 2023-03-20 00:58:12,286 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-20 00:58:12,286 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679273892286"}]},"ts":"9223372036854775807"} 2023-03-20 00:58:12,287 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-20 00:58:12,288 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 00:58:12,289 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 47 msec 2023-03-20 00:58:12,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44395] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-20 00:58:12,345 INFO [Listener at localhost.localdomain/43361] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-03-20 00:58:12,361 INFO [Listener at localhost.localdomain/43361] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=230 (was 227) - Thread LEAK? -, OpenFileDescriptor=561 (was 551) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=576 (was 532) - SystemLoadAverage LEAK? -, ProcessCount=166 (was 168), AvailableMemoryMB=4753 (was 4980) 2023-03-20 00:58:12,362 INFO [Listener at localhost.localdomain/43361] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-03-20 00:58:12,362 INFO [Listener at localhost.localdomain/43361] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-03-20 00:58:12,362 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6a972b34 to 127.0.0.1:60547 2023-03-20 00:58:12,362 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:58:12,363 DEBUG [Listener at localhost.localdomain/43361] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-03-20 00:58:12,363 DEBUG [Listener at localhost.localdomain/43361] util.JVMClusterUtil(257): Found active master hash=253836928, stopped=false 2023-03-20 00:58:12,363 INFO [Listener at localhost.localdomain/43361] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase20.apache.org,44395,1679273724282 2023-03-20 00:58:12,363 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-20 00:58:12,364 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-20 00:58:12,364 INFO [Listener at localhost.localdomain/43361] procedure2.ProcedureExecutor(629): Stopping 2023-03-20 00:58:12,364 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-20 00:58:12,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-20 00:58:12,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-20 00:58:12,366 DEBUG [Listener at localhost.localdomain/43361] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x09c7f582 to 127.0.0.1:60547 2023-03-20 00:58:12,366 DEBUG [Listener at localhost.localdomain/43361] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:58:12,366 INFO [Listener at localhost.localdomain/43361] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,34553,1679273725199' ***** 2023-03-20 00:58:12,366 INFO [Listener at localhost.localdomain/43361] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-03-20 00:58:12,367 INFO [RS:0;jenkins-hbase20:34553] regionserver.HeapMemoryManager(220): Stopping 2023-03-20 00:58:12,367 INFO [RS:0;jenkins-hbase20:34553] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-03-20 00:58:12,367 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-03-20 00:58:12,367 INFO [RS:0;jenkins-hbase20:34553] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-03-20 00:58:12,367 INFO [RS:0;jenkins-hbase20:34553] regionserver.HRegionServer(3304): Received CLOSE for eae49a7cbb97a0b15515bb68f94a4bb9 2023-03-20 00:58:12,367 INFO [RS:0;jenkins-hbase20:34553] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:12,368 DEBUG [RS:0;jenkins-hbase20:34553] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x01e536ae to 127.0.0.1:60547 2023-03-20 00:58:12,368 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing eae49a7cbb97a0b15515bb68f94a4bb9, disabling compactions & flushes 2023-03-20 00:58:12,368 DEBUG [RS:0;jenkins-hbase20:34553] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:58:12,368 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1679273727640.eae49a7cbb97a0b15515bb68f94a4bb9. 2023-03-20 00:58:12,368 INFO [RS:0;jenkins-hbase20:34553] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-03-20 00:58:12,368 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1679273727640.eae49a7cbb97a0b15515bb68f94a4bb9. 2023-03-20 00:58:12,369 INFO [RS:0;jenkins-hbase20:34553] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-03-20 00:58:12,369 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1679273727640.eae49a7cbb97a0b15515bb68f94a4bb9. after waiting 0 ms 2023-03-20 00:58:12,369 INFO [RS:0;jenkins-hbase20:34553] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-03-20 00:58:12,369 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1679273727640.eae49a7cbb97a0b15515bb68f94a4bb9. 2023-03-20 00:58:12,369 INFO [RS:0;jenkins-hbase20:34553] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-03-20 00:58:12,369 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing eae49a7cbb97a0b15515bb68f94a4bb9 1/1 column families, dataSize=78 B heapSize=488 B 2023-03-20 00:58:12,369 INFO [RS:0;jenkins-hbase20:34553] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-03-20 00:58:12,370 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-20 00:58:12,370 DEBUG [RS:0;jenkins-hbase20:34553] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, eae49a7cbb97a0b15515bb68f94a4bb9=hbase:namespace,,1679273727640.eae49a7cbb97a0b15515bb68f94a4bb9.} 2023-03-20 00:58:12,370 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-20 00:58:12,370 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-20 00:58:12,370 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-20 00:58:12,370 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-20 00:58:12,370 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.94 KB heapSize=36.31 KB 2023-03-20 00:58:12,371 DEBUG [RS:0;jenkins-hbase20:34553] regionserver.HRegionServer(1505): Waiting on 1588230740, eae49a7cbb97a0b15515bb68f94a4bb9 2023-03-20 00:58:12,378 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-03-20 00:58:12,389 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.29 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/meta/1588230740/.tmp/info/896f37befe7f4c3f967326e476f03fc3 2023-03-20 00:58:12,395 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 896f37befe7f4c3f967326e476f03fc3 2023-03-20 00:58:12,572 DEBUG [RS:0;jenkins-hbase20:34553] regionserver.HRegionServer(1505): Waiting on 1588230740, eae49a7cbb97a0b15515bb68f94a4bb9 2023-03-20 00:58:12,772 DEBUG [RS:0;jenkins-hbase20:34553] regionserver.HRegionServer(1505): Waiting on 1588230740, eae49a7cbb97a0b15515bb68f94a4bb9 2023-03-20 00:58:12,790 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/namespace/eae49a7cbb97a0b15515bb68f94a4bb9/.tmp/info/4518f24596c14d39ac686941ab388c1c 2023-03-20 00:58:12,794 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/namespace/eae49a7cbb97a0b15515bb68f94a4bb9/.tmp/info/4518f24596c14d39ac686941ab388c1c as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/namespace/eae49a7cbb97a0b15515bb68f94a4bb9/info/4518f24596c14d39ac686941ab388c1c 2023-03-20 00:58:12,797 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/namespace/eae49a7cbb97a0b15515bb68f94a4bb9/info/4518f24596c14d39ac686941ab388c1c, entries=2, sequenceid=6, filesize=4.8 K 2023-03-20 00:58:12,797 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for eae49a7cbb97a0b15515bb68f94a4bb9 in 428ms, sequenceid=6, compaction requested=false 2023-03-20 00:58:12,800 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/meta/1588230740/.tmp/rep_barrier/c97629b4a2ac4fdcbac2a565882e3e29 2023-03-20 00:58:12,802 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/namespace/eae49a7cbb97a0b15515bb68f94a4bb9/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-03-20 00:58:12,803 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1679273727640.eae49a7cbb97a0b15515bb68f94a4bb9. 2023-03-20 00:58:12,803 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for eae49a7cbb97a0b15515bb68f94a4bb9: 2023-03-20 00:58:12,803 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1679273727640.eae49a7cbb97a0b15515bb68f94a4bb9. 2023-03-20 00:58:12,805 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c97629b4a2ac4fdcbac2a565882e3e29 2023-03-20 00:58:12,973 DEBUG [RS:0;jenkins-hbase20:34553] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-20 00:58:13,173 DEBUG [RS:0;jenkins-hbase20:34553] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-20 00:58:13,212 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/meta/1588230740/.tmp/table/3a15927a9ea545f28b797316814801b8 2023-03-20 00:58:13,214 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3a15927a9ea545f28b797316814801b8 2023-03-20 00:58:13,215 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/meta/1588230740/.tmp/info/896f37befe7f4c3f967326e476f03fc3 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/meta/1588230740/info/896f37befe7f4c3f967326e476f03fc3 2023-03-20 00:58:13,218 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 896f37befe7f4c3f967326e476f03fc3 2023-03-20 00:58:13,218 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/meta/1588230740/info/896f37befe7f4c3f967326e476f03fc3, entries=22, sequenceid=93, filesize=7.1 K 2023-03-20 00:58:13,219 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/meta/1588230740/.tmp/rep_barrier/c97629b4a2ac4fdcbac2a565882e3e29 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/meta/1588230740/rep_barrier/c97629b4a2ac4fdcbac2a565882e3e29 2023-03-20 00:58:13,222 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c97629b4a2ac4fdcbac2a565882e3e29 2023-03-20 00:58:13,222 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/meta/1588230740/rep_barrier/c97629b4a2ac4fdcbac2a565882e3e29, entries=6, sequenceid=93, filesize=5.5 K 2023-03-20 00:58:13,223 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/meta/1588230740/.tmp/table/3a15927a9ea545f28b797316814801b8 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/meta/1588230740/table/3a15927a9ea545f28b797316814801b8 2023-03-20 00:58:13,225 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3a15927a9ea545f28b797316814801b8 2023-03-20 00:58:13,225 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/meta/1588230740/table/3a15927a9ea545f28b797316814801b8, entries=9, sequenceid=93, filesize=5.5 K 2023-03-20 00:58:13,226 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.94 KB/21446, heapSize ~36.27 KB/37136, currentSize=0 B/0 for 1588230740 in 856ms, sequenceid=93, compaction requested=false 2023-03-20 00:58:13,232 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-03-20 00:58:13,233 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-20 00:58:13,234 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-20 00:58:13,234 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-20 00:58:13,235 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-03-20 00:58:13,243 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-03-20 00:58:13,243 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-03-20 00:58:13,373 INFO [RS:0;jenkins-hbase20:34553] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,34553,1679273725199; all regions closed. 2023-03-20 00:58:13,386 DEBUG [RS:0;jenkins-hbase20:34553] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/oldWALs 2023-03-20 00:58:13,386 INFO [RS:0;jenkins-hbase20:34553] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C34553%2C1679273725199.meta:.meta(num 1679273727385) 2023-03-20 00:58:13,390 DEBUG [RS:0;jenkins-hbase20:34553] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/oldWALs 2023-03-20 00:58:13,390 INFO [RS:0;jenkins-hbase20:34553] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C34553%2C1679273725199:(num 1679273726389) 2023-03-20 00:58:13,390 DEBUG [RS:0;jenkins-hbase20:34553] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:58:13,390 INFO [RS:0;jenkins-hbase20:34553] regionserver.LeaseManager(133): Closed leases 2023-03-20 00:58:13,391 INFO [RS:0;jenkins-hbase20:34553] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-03-20 00:58:13,391 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-20 00:58:13,392 INFO [RS:0;jenkins-hbase20:34553] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:34553 2023-03-20 00:58:13,396 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-20 00:58:13,396 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,34553,1679273725199 2023-03-20 00:58:13,396 ERROR [Listener at localhost.localdomain/43361-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5f23405 rejected from java.util.concurrent.ThreadPoolExecutor@1cda063[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2001] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-20 00:58:13,398 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-20 00:58:13,398 ERROR [Listener at localhost.localdomain/43361-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@610718e7 rejected from java.util.concurrent.ThreadPoolExecutor@1cda063[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2001] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-20 00:58:13,398 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,34553,1679273725199] 2023-03-20 00:58:13,399 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,34553,1679273725199; numProcessing=1 2023-03-20 00:58:13,400 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,34553,1679273725199 already deleted, retry=false 2023-03-20 00:58:13,400 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase20.apache.org,34553,1679273725199 expired; onlineServers=0 2023-03-20 00:58:13,400 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,44395,1679273724282' ***** 2023-03-20 00:58:13,400 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-03-20 00:58:13,400 DEBUG [M:0;jenkins-hbase20:44395] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4e1c34bb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-03-20 00:58:13,400 INFO [M:0;jenkins-hbase20:44395] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,44395,1679273724282 2023-03-20 00:58:13,400 INFO [M:0;jenkins-hbase20:44395] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,44395,1679273724282; all regions closed. 2023-03-20 00:58:13,400 DEBUG [M:0;jenkins-hbase20:44395] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 00:58:13,400 DEBUG [M:0;jenkins-hbase20:44395] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-03-20 00:58:13,400 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-03-20 00:58:13,400 DEBUG [M:0;jenkins-hbase20:44395] cleaner.HFileCleaner(317): Stopping file delete threads 2023-03-20 00:58:13,400 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1679273726122] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1679273726122,5,FailOnTimeoutGroup] 2023-03-20 00:58:13,400 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1679273726124] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1679273726124,5,FailOnTimeoutGroup] 2023-03-20 00:58:13,400 INFO [M:0;jenkins-hbase20:44395] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-03-20 00:58:13,401 INFO [M:0;jenkins-hbase20:44395] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-03-20 00:58:13,401 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-03-20 00:58:13,401 INFO [M:0;jenkins-hbase20:44395] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-03-20 00:58:13,401 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-20 00:58:13,401 DEBUG [M:0;jenkins-hbase20:44395] master.HMaster(1512): Stopping service threads 2023-03-20 00:58:13,402 INFO [M:0;jenkins-hbase20:44395] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-03-20 00:58:13,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-20 00:58:13,402 ERROR [M:0;jenkins-hbase20:44395] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] Thread[IPC Client (1465462354) connection to localhost.localdomain/127.0.0.1:35057 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-03-20 00:58:13,402 INFO [M:0;jenkins-hbase20:44395] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-03-20 00:58:13,402 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-03-20 00:58:13,403 DEBUG [M:0;jenkins-hbase20:44395] zookeeper.ZKUtil(398): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-03-20 00:58:13,403 WARN [M:0;jenkins-hbase20:44395] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-03-20 00:58:13,403 INFO [M:0;jenkins-hbase20:44395] assignment.AssignmentManager(315): Stopping assignment manager 2023-03-20 00:58:13,403 INFO [M:0;jenkins-hbase20:44395] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-03-20 00:58:13,404 DEBUG [M:0;jenkins-hbase20:44395] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-20 00:58:13,404 INFO [M:0;jenkins-hbase20:44395] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-20 00:58:13,404 DEBUG [M:0;jenkins-hbase20:44395] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-20 00:58:13,404 DEBUG [M:0;jenkins-hbase20:44395] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-20 00:58:13,404 DEBUG [M:0;jenkins-hbase20:44395] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-20 00:58:13,404 INFO [M:0;jenkins-hbase20:44395] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.09 KB heapSize=393.73 KB 2023-03-20 00:58:13,500 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-20 00:58:13,500 INFO [RS:0;jenkins-hbase20:34553] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,34553,1679273725199; zookeeper connection closed. 2023-03-20 00:58:13,500 ERROR [Listener at localhost.localdomain/43361-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@196322ad rejected from java.util.concurrent.ThreadPoolExecutor@1cda063[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2001] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-20 00:58:13,501 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5aeba3ba] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5aeba3ba 2023-03-20 00:58:13,501 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): regionserver:34553-0x1002c399e9c0001, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-20 00:58:13,501 ERROR [Listener at localhost.localdomain/43361-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7d79252a rejected from java.util.concurrent.ThreadPoolExecutor@1cda063[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2001] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-20 00:58:13,502 INFO [Listener at localhost.localdomain/43361] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-03-20 00:58:13,813 INFO [M:0;jenkins-hbase20:44395] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.09 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/fecdc681c851457898c5f62486043577 2023-03-20 00:58:13,822 INFO [M:0;jenkins-hbase20:44395] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for fecdc681c851457898c5f62486043577 2023-03-20 00:58:13,824 DEBUG [M:0;jenkins-hbase20:44395] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/fecdc681c851457898c5f62486043577 as hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/fecdc681c851457898c5f62486043577 2023-03-20 00:58:13,828 INFO [M:0;jenkins-hbase20:44395] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for fecdc681c851457898c5f62486043577 2023-03-20 00:58:13,828 INFO [M:0;jenkins-hbase20:44395] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35057/user/jenkins/test-data/03481cd8-4311-47b3-c62d-fffbd72082b6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/fecdc681c851457898c5f62486043577, entries=65, sequenceid=578, filesize=24.7 K 2023-03-20 00:58:13,829 INFO [M:0;jenkins-hbase20:44395] regionserver.HRegion(2948): Finished flush of dataSize ~345.09 KB/353372, heapSize ~393.71 KB/403160, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 425ms, sequenceid=578, compaction requested=false 2023-03-20 00:58:13,830 INFO [M:0;jenkins-hbase20:44395] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-20 00:58:13,830 DEBUG [M:0;jenkins-hbase20:44395] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-20 00:58:13,833 INFO [M:0;jenkins-hbase20:44395] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-03-20 00:58:13,833 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-20 00:58:13,834 INFO [M:0;jenkins-hbase20:44395] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:44395 2023-03-20 00:58:13,835 DEBUG [M:0;jenkins-hbase20:44395] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,44395,1679273724282 already deleted, retry=false 2023-03-20 00:58:13,937 INFO [M:0;jenkins-hbase20:44395] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,44395,1679273724282; zookeeper connection closed. 2023-03-20 00:58:13,937 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-20 00:58:13,938 ERROR [Listener at localhost.localdomain/43361-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6f86f09f rejected from java.util.concurrent.ThreadPoolExecutor@723ddd23[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3349] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-20 00:58:13,938 DEBUG [Listener at localhost.localdomain/43361-EventThread] zookeeper.ZKWatcher(600): master:44395-0x1002c399e9c0000, quorum=127.0.0.1:60547, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-20 00:58:13,938 ERROR [Listener at localhost.localdomain/43361-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2d2bc66d rejected from java.util.concurrent.ThreadPoolExecutor@723ddd23[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3349] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-20 00:58:13,941 WARN [Listener at localhost.localdomain/43361] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-03-20 00:58:13,946 INFO [Listener at localhost.localdomain/43361] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-20 00:58:13,967 WARN [BP-583762141-148.251.75.209-1679273721698 heartbeating to localhost.localdomain/127.0.0.1:35057] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-03-20 00:58:13,967 WARN [BP-583762141-148.251.75.209-1679273721698 heartbeating to localhost.localdomain/127.0.0.1:35057] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-583762141-148.251.75.209-1679273721698 (Datanode Uuid 14d682db-b1e5-40dd-85c3-3fd110b80fa0) service to localhost.localdomain/127.0.0.1:35057 2023-03-20 00:58:13,970 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/39532ada-9017-8fdd-ef0b-a0e859f23e00/cluster_4c1651d3-7e91-3b9e-86c3-2dd8b012a0ed/dfs/data/data1/current/BP-583762141-148.251.75.209-1679273721698] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-20 00:58:13,970 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/39532ada-9017-8fdd-ef0b-a0e859f23e00/cluster_4c1651d3-7e91-3b9e-86c3-2dd8b012a0ed/dfs/data/data2/current/BP-583762141-148.251.75.209-1679273721698] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-20 00:58:14,003 INFO [Listener at localhost.localdomain/43361] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-03-20 00:58:14,159 INFO [Listener at localhost.localdomain/43361] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-03-20 00:58:14,323 INFO [Listener at localhost.localdomain/43361] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-03-20 00:58:14,406 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(152): Removing adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint